Windows Server 2012 NIC Teaming and Multichannel

Notes from TechEd NA 2012 WSV314:

Terminology

• It is a Team, not NIC bonding, etc.
• A team is made of Team Members
• Team Interfaces are the virtual NICs that can connect to a team and have IP stacks, etc.  You can call them tNICs to differentiate them from vNICs in the Hyper-V world.

Team Connection Modes

Most people don’t know the teaming mode they select when using OEM products.  MSFT are clear about what teaming does under the cover.  Connection mode = how do you connect to the switch?

• Switch Independent can be used where the switch doesn’t need to know anything about the team.
• Switch dependent teaming is when the switch does need to know something about the team. The switch decides where to send the inbound traffic.

There are 2 switch dependent modes:

• LACP (Link Aggregation Control Protocol) is where the is where the host and switch agree on who the team members are. IEEE 802.1ax
• Static Teaming is where you configure it on the switch.

Load Distribution Modes

You also need to know how you will spread traffic across the team members in the team.

1) Address Hash comes in 3 flavours:

• 4-tuple (the default): Uses RSS on the TCP/UDP ports. 
• 2-tuple: If the ports aren’t available (encrypted traffic such as IPsec) then it’ll go to 2-tuple where it uses the IP address.
• MAC address hash: If not IP traffic, then MAC addresses are hashed.

2) We also have Hyper-V Port, where it hashes the port number on the Hyper-V switch that the traffic is coming from.  Normally this equates to per-VM traffic.  No distribution of traffic.  It maps a VM to a single NIC.  If a VM needs more pipe than a single NIC can handle then this won’t be able to do it.  Shouldn’t be a problem because we are consolidating after all.

Maybe create a team in the VM?  Make sure the vNICs are on different Hyper-V Switches. 

SR-IOV

Remember that SR-IOV bypasses the host stack and therefore can’t be teamed at the host level.  The VM bypasses it.  You can team two SR-IOV enabled vNICs in the guest OS for LBFO.

Switch Independent – Address Hash

Outbound traffic in Address Hashing will spread across NICs. All inbound traffic is targeted at a single inbound MAC address for routing purposes, and therefore only uses 1 NIC.  Best used when:

• Switch diversity is a concern
• Active/Standby mode
• Heavy outbound but light inbound workloads

Switch Independent – Hyper-V Port

All traffic from each VM is sent out on that VM’s physical NIC or team member.  Inbound traffic also comes in on the same team member.  So we can maximise NIC bandwidth.  It also allows for maximum use of VMQs for better virtual networking performance.

Best for:

• Number of VMs well exceeds number of team members
• You’re OK with VM being restricted to bandwidth of a single team member

Switch Dependent Address Hash

Sends on all active members by using one of the hashing methods.  Receives on all ports – the switch distributes inbound traffic.  No association between inbound and outbound team members.  Best used for:

• Native teaming for maximum performance and switch diversity is not required.
• Teaming under the Hyper-V switch when a VM needs to exceed the bandwidth limits of a single team member  Not as efficient with VMQ because we can’t predict the traffic.

Best performance for both inbound and outbound.

Switch Dependent – Hyper-V Port

Sends on all active members using the hashed port – 1 team member per VM.  Inbound traffic is distributed by the switch  on all ports so there is no correlation to inbound and outbound.  Best used when:

• When number of VMs on the switch well exceeds the number of team members AND
• You have a policy that says you must use switch dependent teaming.

When using Hyper-V you will normally want to use Switch Independent & Hyper-V Port mode. 

When using native physical servers you’ll likely want to use Switch Independent & Address Hash.  Unless you have a policy that can’t tolerate a switch failure.

Team Interfaces

There are different ways of interfacing with the team:

• Default mode: all traffic from all VLANs is passed through the team
• VLAN mode: Any traffic that matches a VLAN ID/tag is passed through.  Everything else is dropped.

Inbound traffic passes through to one team interface at once.

The only supported configuration for Hyper-V is shown above: Default mode passing through all traffic t the Hyper-V Switch.  Do all the VLAN tagging and filtering on the Hyper-V Switch.  You cannot mix other interfaces with this team – the team must be dedicated to the Hyper-V Switch.  REPEAT: This is the only supported configuration for Hyper-V.

A new team has one team interface by default. 

Any team interfaces created after the initial team creation must be VLAN mode team interfaces (bound to a VLAN ID).  You can delete these team interfaces.

Get-NetAdapter: Get the properties of a team interface

Rename-NetAdapter: rename a team interface

Team Members

• Any physical ETHERNET adapter with a Windows Logo (for stability reasons and promiscuous mode for VLAN trunking) can be a team member.
• Teaming of InfiniBand, Wifi, WWAN not supported.
• Teams made up of teams not supported.

You can have team members in active or standby mode.

Virtual Teams

Supported if:

• No more than 2 team members in the guest OS team

Notes:

• Intended for SR-IOV NICs but will work without it.
• Both vNICs in the team should be connected to different virtual switches on different physical NICs

If you try to team a vNIC that is not on an External switch, it will show up fine and OK until you try to team it.  Teaming will shut down the vNIC at that point. 

You also have to allow teaming in a vNIC in Advanced Properties – Allow NIC teaming.  Do this for each of the VM’s vNICs.  Without this, failover will not succeed. 

PowerShell CMDLETs for Teaming

The UI is actually using POSH under the hood.  You can use the NIC Teaming UI to remotely manage/configure a server using RSAT for Windows 8.  WARNING: Your remote access will need to run over a NIC that you aren’t altering because you would lose connectivity.

Supported Networking Features

NIC teaming works with almost everything:

TCP Chimney Offload, RDMA and SR-IOV bypass the stack so obviously they cannot be teamed in the host.

Limits

• 32 NICs in a team
• 32 teams
• 32 team interfaces in a team

That’s a lot of quad port NICs.  Good luck with that!  

SMB Multichannel

An alternative to a team in an SMB 3.0 scenario.  Can use multiple NICs with same connectivity, and use multiple cores via NIC RSS to have simultaneous streams over a single NIC (RSS) or many NICs (teamed, not teamed, and also with RSS if available).  Basically, leverage more bandwidth to get faster SMB 3.0 throughput.

Without it, a 10 GbE NIC would only be partly used by SMB – single CPU core trying to transmit.  RSS makes it multi-threaded/core, and therefore many connections by the data transfer.

Remember – you cannot team RDMA.  So another case to use Multichannel and get an LBFO effect is to use SMB Multichannel …. or I should say “use” … SMB 3.0 turns it on automatically if multiple paths are available between client and server.

SMB 3.0 is NUMA aware.

Multichannel will only use NICs of same speed/type.  Won’t see traffic spread over a 10 GbE and a 1 GbE NIC, for example, or over RDMA-enabled and non-RDMA NICs. 

In tests, the throughput on RSS enabled 10 GbE NICs (1, 2, 3, and 4 NICs), seemed to grow in a predictable near-linear rate.

SMB 3.0 uses a shortest queue first algorithm for load balancing – basic but efficient.

SMB Multichannel and Teaming

Teaming allows for faster failover.  MSFT recommending teaming where applicable.  Address-hash port mode with Multichannel can be a nice solution.  Multichannel will detect a team and create multiple connections over the team.

RDMA

If RDMA is possible on both client and server then SMB 3.0 switches over to SMB Direct.  Net monitoring will see negotiation, and then … “silence” for the data transmission.  Multichannel is supported across single or multiple NICs – no NIC teaming, remember!

Won’t Work With Multichannel

• Single non-RSS capable NIC
• Different type/speed NICs, e.g. 10 GbE RDMA favoured over 10 GbE non-RDMA NIC
• Wireless can be failed from but won’t be used in multi-channel

Supported Configurations

Note that Multichannel over a team of NICs is favoured over multichannel over the same NICs that are not in a team.  Added benefits of teaming (types, and fast failover detection).  This applies, whether the NICs are RSS capable or not.  And the team also benefits non-SMB 3.0 traffic.

Troubleshooting SMB Multichannel

Plenty to think about there, folks!  Where it applies in Hyper-V?

• NIC teaming obviously applies.
• Multichannel applies in the cluster: redirected IO over the cluster communications network
• Storing VMs on SMB 3.0 file shares