Enabling SMB Multichannel On Scale-Out File Server Cluster Nodes

Ten days ago I highlighted a blog post by Microsoft’s Jose Baretto that SMB Multichannel across multiple NICs in a clustered node required that both NICs be in different subnets.  That means:

  • You have 2 NICs in each node in the Scale-Out File Server cluster
  • Both NICs must be in different subnets
  • You must enable both NICs for client access
  • There will be 2 NICs in each of the hosts that are also on these subnets, probably dedicated to SMB 3.0 comms, depending on if/how you do converged fabrics

image

You can figure out cabling and IP addressing for yourself – if not, you need to not be doing this work!

The question is, what else must you do?  Well, SMB Multichannel doesn’t need any configuration to work.  Pop the NICs into the Hyper-V hosts and away you go.  On the SOFS cluster, there’s a little bit more work.

After you create the SOFS cluster, you need to make sure that client communications is enabled on both of the NICs on subnet 1 and subnet 2 (as above).  This is to allow the Hyper-V hosts to talk to the SOFS across both NICs (the green NICs in the diagram) in the SOFS cluster nodes.  You can see this setting below.  In my demo lab, my second subnet is not routed and it wasn’t available to configure when I created the SOFS cluster.

image

You’ll get a warning that you need to enable a Client Access Point (with an IP address) for the cluster to accept communications on this network.  Damned if I’ve found a way to do that.  I don’t think it’s necessary to do that additional step in the case of an SOFS, as you’ll see in a moment.  I’ll try to confirm that with MSFT.  Ignore the warning and continue.  My cluster (uses iSCSI because I don’t have a JBOD) looks like:

image

You can see ManagementOS1 and ManagementOS2 (on different subnets) are Enabled, meaning that I’ve allowed clients to connect through both networks.  ManagementOS1 has the default CAP (configured when the cluster was created).

Next I created the file server for application data role (aka the SOFS).  Over in AD we find a computer object for the SOFS and we should see that 4 IP addresses have been registered in DNS.  Note how the SOFS role uses the IP addresses of the SOFS cluster nodes (demo-fs1 and demo-fs2).  You can also need the DNS records for my 2 hosts (on 2 subnets) here.

image

If you don’t see 2 IP address for each SOFS node registered with the SOFS name (as above – 2 addresses * 2 nodes = 4) then double check that you have enabled client communications across both cluster networks for the NICs on the SOFS cluster nodes (as previous).

Now we should be all ready to rock and role.

In my newly modified demo lab, I run this with the hosts clustered (to show new cluster Live Migration features) and not clustered (to show Live Migration with SMB storage).  The eagle-eyed will notice that my demo Hyper-V hosts don’t have dedicated NICs for SMB comms.  In the real world, I’d probably have dedicated NICs for SMB 3.0 comms on the Hyper-V hosts.  They’d be on the 2 subnets that have been referred to in this post.

Please follow and like us:

23 Comments on Enabling SMB Multichannel On Scale-Out File Server Cluster Nodes

  1. I tried this setup with Infinband for RDMA and this setup does not work. Does the RDMA network need to be routable to my domain?

  2. Yes, I have but MS tells me that my RDMA Infiniband interfaces do not have to be routable to my domain.

    • And they would be right. Have a search for the step-by-step instructions for Infiniband SOFS by Jose Barreto. Mellanox didn’t see fit to give me gear for my lab so I cannot advise on Infiniband.

  3. However, for clustered SOFS the RDMA adapters need to be on separate IP segments, each RDMA adapter needs to be able to route to the other & registered in DNS.

    • No routing required AFAIK. And you only need to put in the DNS config on the storage NICs to force a registration (which goes via the management NICs).

  4. BTW, Jose Barreto’s blog only shows a single node SOFS, not a cluster. I used this as a reference for my design.

  5. RookieSOFSUser // December 6, 2014 at 9:41 PM // Reply

    I am having a similar issue.
    Each of my SOFS node have 1 Gb mgmt nic – default CAP and main link to AD/DNS.
    Each of my SOFS nodes and Hyper-V host also have dual port Mellenox RDMA nics for SMB comm. I setup separate subnet on each of these RDMA nics for SMB multichannel. These subnets are non-routable to the rest of the network AD or DNS, as I only want my Hyper-V hosts to talk to this separate storage network. I setup CAP on the RDMA nics. I assumed the servers would just work with SMB multichannel.My performance is terrible.
    Do all CAPs /RDMA nics need to register in DNS to function properly?

    • Yes. They don’t need networking to it; the servers will register the rNICs via the routed NICs.

      • I have the same problem, and same setup.
        Exactly how will they register the rNICs in the DNS? How do you configure the Routing?
        I get great performance if I do \\fileserver\share, since the host will communnicate over rNICs, but if I try \\FileClusterName\Share it will ask the DNS and use the mgmt NIC.

        • Simply configure DNS on the rNICs. The registration goes via the management NIC.

          • Thanks! I could have sworn somewhere I read not to enable DNS registration on those rNICs. It works perfect now. Do you recommend using the ExcludeNetwork to remove the MGMT NICs from DNS? If not, what happens when a host hits the MGMT NIC during round robin?

          • No prob! SMB Multichannel will select the rNICs because RDMA trumps non-RDMA NICs :) Do enable SMB Multichannel Constraints on all nodes (SOFS and hosts) to restrict it to the rNICs to enforce this.

      • Okay, I just added the DNS server on the rNICs and now they’re registered in DNS.
        However the problem persists:
        \\FileClusterName\Share still use mgmt nic.

        • Have you tested when SMB mutlichannel has kicked in? And then read this: https://www.petri.com/controlling-smb-multichannel

        • On both Chelsio rNICs I have configured an IP address, Subnet and DNS. I have disabled “register in DNS” and NetBIOS over TCP/IP on the rNICs but still they do not register in DNS. The rNICs have the same DNS settings as the MGMT teamed NIC. I also have the rNICs set for client communications. I have the MGMT teamed NIC at the top of the binding order followed by the two rNICs. I bounced the SOFS role more times than I can count. Any thoughts?

  6. You mentioned for real world use you would rather have dedicated SMB 3 NICs on your Hyper-V cluster nodes. With redundant switches and converged networking, what value is added if bandwidth wasn’t an issue?

    Is it more of a safety net should the vSwitch have an issue?

  7. how about using good old “hosts” file on the HV hoster so they access the SMB not-routable VLAN while the DNS keps pointing – as the set up dictated to the routable mang vlan of the servers?
    this way when you try to remote to sofs1 name from mng network u get the active sofs cluster member yet the HV hosters will resolve to the SMB network.

    but maybe the round robin will be broken this way?

    so perhaps the oposit and use hosts on my mng station so i access the correct role holder?

    • Use DNS as prescribed. Those of you who venture out into unknown country end up being eaten by wolves.

  8. Hi Finn, hi have a 2 node sofs cluster with a teamed nic (without rdma). Do i have to bind 3 networks on this nic ?

    -smb1 ( smb1, live mig.)
    -smb2 (smb2, CSV)
    -mgmt (management)

    regards
    Mario

Leave a comment

Your email address will not be published.

*