Ten days ago I highlighted a blog post by Microsoft’s Jose Baretto that SMB Multichannel across multiple NICs in a clustered node required that both NICs be in different subnets.  That means:

  • You have 2 NICs in each node in the Scale-Out File Server cluster
  • Both NICs must be in different subnets
  • You must enable both NICs for client access
  • There will be 2 NICs in each of the hosts that are also on these subnets, probably dedicated to SMB 3.0 comms, depending on if/how you do converged fabrics


You can figure out cabling and IP addressing for yourself – if not, you need to not be doing this work!

The question is, what else must you do?  Well, SMB Multichannel doesn’t need any configuration to work.  Pop the NICs into the Hyper-V hosts and away you go.  On the SOFS cluster, there’s a little bit more work.

After you create the SOFS cluster, you need to make sure that client communications is enabled on both of the NICs on subnet 1 and subnet 2 (as above).  This is to allow the Hyper-V hosts to talk to the SOFS across both NICs (the green NICs in the diagram) in the SOFS cluster nodes.  You can see this setting below.  In my demo lab, my second subnet is not routed and it wasn’t available to configure when I created the SOFS cluster.


You’ll get a warning that you need to enable a Client Access Point (with an IP address) for the cluster to accept communications on this network.  Damned if I’ve found a way to do that.  I don’t think it’s necessary to do that additional step in the case of an SOFS, as you’ll see in a moment.  I’ll try to confirm that with MSFT.  Ignore the warning and continue.  My cluster (uses iSCSI because I don’t have a JBOD) looks like:


You can see ManagementOS1 and ManagementOS2 (on different subnets) are Enabled, meaning that I’ve allowed clients to connect through both networks.  ManagementOS1 has the default CAP (configured when the cluster was created).

Next I created the file server for application data role (aka the SOFS).  Over in AD we find a computer object for the SOFS and we should see that 4 IP addresses have been registered in DNS.  Note how the SOFS role uses the IP addresses of the SOFS cluster nodes (demo-fs1 and demo-fs2).  You can also need the DNS records for my 2 hosts (on 2 subnets) here.


If you don’t see 2 IP address for each SOFS node registered with the SOFS name (as above – 2 addresses * 2 nodes = 4) then double check that you have enabled client communications across both cluster networks for the NICs on the SOFS cluster nodes (as previous).

Now we should be all ready to rock and role.

In my newly modified demo lab, I run this with the hosts clustered (to show new cluster Live Migration features) and not clustered (to show Live Migration with SMB storage).  The eagle-eyed will notice that my demo Hyper-V hosts don’t have dedicated NICs for SMB comms.  In the real world, I’d probably have dedicated NICs for SMB 3.0 comms on the Hyper-V hosts.  They’d be on the 2 subnets that have been referred to in this post.

15 comments so far

Add Your Comment
  1. I tried this setup with Infinband for RDMA and this setup does not work. Does the RDMA network need to be routable to my domain?

    • Have you configured DNS settings for the storage NICs?

  2. Yes, I have but MS tells me that my RDMA Infiniband interfaces do not have to be routable to my domain.

    • And they would be right. Have a search for the step-by-step instructions for Infiniband SOFS by Jose Barreto. Mellanox didn’t see fit to give me gear for my lab so I cannot advise on Infiniband.

  3. However, for clustered SOFS the RDMA adapters need to be on separate IP segments, each RDMA adapter needs to be able to route to the other & registered in DNS.

    • No routing required AFAIK. And you only need to put in the DNS config on the storage NICs to force a registration (which goes via the management NICs).

  4. BTW, Jose Barreto’s blog only shows a single node SOFS, not a cluster. I used this as a reference for my design.

  5. I am having a similar issue.
    Each of my SOFS node have 1 Gb mgmt nic – default CAP and main link to AD/DNS.
    Each of my SOFS nodes and Hyper-V host also have dual port Mellenox RDMA nics for SMB comm. I setup separate subnet on each of these RDMA nics for SMB multichannel. These subnets are non-routable to the rest of the network AD or DNS, as I only want my Hyper-V hosts to talk to this separate storage network. I setup CAP on the RDMA nics. I assumed the servers would just work with SMB multichannel.My performance is terrible.
    Do all CAPs /RDMA nics need to register in DNS to function properly?

    • Yes. They don’t need networking to it; the servers will register the rNICs via the routed NICs.

  6. You mentioned for real world use you would rather have dedicated SMB 3 NICs on your Hyper-V cluster nodes. With redundant switches and converged networking, what value is added if bandwidth wasn’t an issue?

    Is it more of a safety net should the vSwitch have an issue?

    • Two reasons: RSS and RDMA cannot be used in virtual NICs in the management OS.

  7. how about using good old “hosts” file on the HV hoster so they access the SMB not-routable VLAN while the DNS keps pointing – as the set up dictated to the routable mang vlan of the servers?
    this way when you try to remote to sofs1 name from mng network u get the active sofs cluster member yet the HV hosters will resolve to the SMB network.

    but maybe the round robin will be broken this way?

    so perhaps the oposit and use hosts on my mng station so i access the correct role holder?

    • Use DNS as prescribed. Those of you who venture out into unknown country end up being eaten by wolves.

  8. Hi Finn, hi have a 2 node sofs cluster with a teamed nic (without rdma). Do i have to bind 3 networks on this nic ?

    -smb1 ( smb1, live mig.)
    -smb2 (smb2, CSV)
    -mgmt (management)


    • What speed and how many NICs do you have on the NIC team? How many NICs do the SOFS nodes have? The design I recommend is here: http://www.aidanfinn.com/?p=14879

Get Adobe Flash player