2013
01.10

Windows Server 2012 NIC Teaming Part 1 – Back To Basics

Windows Server 2012 NIC Teaming Part 2 – What’s What?

Windows Server 2012 NIC Teaming Part 3 – Switch Connection Modes

With a team formed you have the failover part of LBFO figured out.  But what about the load balancing piece of LBFO?  That’s what this post is going to discuss.

First, think about some concepts:

  • A packet sent into the NIC team should not be fragmented and sent across multiple NICs.  We like BIG packets because they fill bandwidth and reduce the time to get data from A to B.
  • Sometimes we need to make the path of traffic predictable … very predictable.  And sometimes we don’t … but there still needs to be some organisation.

There are 2 traffic or load distribution algorithms in WS2012 NIC teaming (actually it’s more if you dig into it).  The one you choose when creating/configuring a team depends on the traffic and the purpose of the team.

Hyper-V Switch Port

Generally speaking, this is the load distribution that you should use when creating a NIC team that will be used to connect a Hyper-V external virtual switch to the LAN, as below.

image

However, you do not have to choose this type of load distribution for this architecture, but it is my rule of thumb.  Let’s get into specifics.

Hyper-V Switch Port will route traffic from a virtual NIC (either in a VM or in the management OS) to a single physical NIC in the team (a team member).  Let’s illustrate that.  In the below diagram, the NIC team is associating the traffic from the vNIC in VM01 with the team member called pNIC1.  The traffic from the vNIC in VM02 is being sent to pNIC2.  Two things to note:

  • The traffic path is predictable (unless a team member fails).  Incoming traffic to the virtual NICs is also going to flow through their associated physical NICs
  • This is not a per-VM association.  It is a per-virtual NIC association.  If we add a second virtual NIC to VM01 then the traffic for that virtual NIC could be associated with any team member by the NIC team.

image

This is one of the things that can confuse people.  They see a team of 2 NICs, maybe giving them a “2 Gbps” or “20 Gbps” pipe.  True, there is a total aggregation of bandwidth, but access to that bandwidth is given on per-team member basis.  That means the virtual NIC in VM02 cannot exceed 1 Gbps or 10 Gbps, depending on the speeds of the team members (physical NICs in the team).

Hyper-V Switch Port is appropriate if the team is being used for an external virtual switch (like the above examples) and:

  • You have more virtual NICs than you have physical NICs.  Maybe you have 2 physical NICs and 20 virtual machines.  Maybe you have 2 physical NICs and you are creating a converged fabric design with 4 virtual NICs in the management OS and several virtual machines.
  • You plan on using the Dynamic Virtual Machine Queue (DVMQ) hardware offload then you should use Hyper-V Switch Port traffic distribution.  DVMQ uses an RSS queue device in a team member to accelerate inbound traffic to a virtual NIC.  The RSS queue must be associated with the virtual NIC and that means the path of inbound traffic must come through the same team member every time… and Hyper-V Switch Port happens to do this via the association process.

As I said, there are times, when you might not use Hyper-V Switch Port.  Maybe you have some massive host, and you’re going to have just 2 massive VMs on it.  You could use one of the alternative load distribution algorithms then.  But that’s a very rare scenario.  I like to keep it simple for people: use Hyper-V Switch Port if you are creating the NIC team for a Hyper-V external virtual switch … unless you understand what’s going on under the hood and have one of those rare situations to vary.

Address Hashing

This method of traffic distribution in the NIC team does not associate virtual NICs with team members.  Instead, each packet that is sent down to the NIC team by the host/server is inspected.  The destination details of the packet (which can include MAC address, IP address, and port numbers) are inspected by the team to determine which team member to send the packet to.

You can see an example of this in the below diagram.  VM01 is sending 2 packets, one to address A and the other to address B.  The NIC team receives the packets, performs a hashing algorithm (hence the name Address Hashing) on the destination details, and uses the results to determine the team member (physical NIC) that will send each packet.  In this case, the packet being send to A goes via pNIC1 and the packet being sent to B is going via pNIC2.

image

In theory, this means that a virtual NIC can take advantage of all the available bandwidth in the NIC team, e.g. the full 2 Gbps or 20 Gbps.  But this is completely dependent on the results of the hashing algorithm.   Using the above example, if all data is going to address A, then all packets will travel through pNIC1.

And that brings us to a most common question about NIC teams and bandwidth.  Say I have a host (or any server) that uses a nice big fat 20 GbE NIC team for Live Migration (or any traffic of a specific protocol).  I want to test Live Migration and the NIC team.  I pause the host, open up PerfMon and expect to see Live Migration using up all 20 Gbps of my NIC team.  What is going on here, under the hood?

  • Host1 is sending data to the single IP address of Host2 on the Live Migration network.
  • Live Migration is sending packets down to the NIC team.  The NIC team inspects each packet, and every one of them has the same destination details: the same MAC address, the same IP address, and same TCP port on Host2.
  • The destination details are hashed and result in all of the packets being sent via a single team member, pNIC1 in this case (see the below figure).
  • This limits Live Migration to the bandwidth of a single team member in the team.

image

That doesn’t mean Live Migration (or any other protocol – I just picked Live Migration because that’s the one Hyper-V engineers are likely to test with first) is limited to just a single team member.  Maybe I have a 3rd host, Host3, and pausing Host1 will cause VMs to Live Migrate to both Host2 and Host3.  The resulting hashing of destination addresses might cause the NIC team to use both team members in Host1 and give me a much better chance at fully using my lovely 20 GbE NIC team (other factors impact bandwidth utilization by Live Migration).

image

A misconception of Address Hashing is that packet1 to addressA will go via teamMember1, and pack2 to the same address (addressA) will go via teamMember2.  I have shown that this is not the case.  However, in most situations, traffic is going to all sorts of addresses and ports, and over the long term you should see different streams of traffic balancing across all of the team members in the NIC team … unless you have a 2 node Hyper-V cluster and are focusing on comms between the two hosts.  In that case, you’ll see 50% utilization of a 2-team-member NIC team – and you’ll be getting the FO part of LBFO until you add a third host.

If you configure a team in the GUI, you are only going to see Hyper-V Switch Port or Address Hash as the Load Balancing Mode options.  Using PowerShell, however, and you can be very precise about the type of Address Hashing that you want to do.  Note that the GUI “Address Hashing” option will use these in order of preference depending on the packets:

  1. TransportPorts (4-Tuple Hash): Uses the source and destination UDP/TCP ports and the IP addresses to create a hash and then assigns the packets that have the matching hash value to one of the available interfaces.
  2. IPAddresses (2-Tuple Hash): Uses the source and destination IP addresses to create a hash and then assigns the packets that have the matching hash value to one of the available interfaces. Used when the traffic is not UDP- or TCP-based, or that detail is hidden (such as with IPsec)
  3. MacAddresses: Uses the source and destination MAC addresses to create a hash and then assigns the packets that have the matching hash value to one of the available interfaces.  Used when the traffic is not IP-based.

My rule of thumb for Address Hashing is that I’ll use it for NIC teams that are nothing to do with a Hyper-V virtual switch, such as a NIC team in a non-host server, or a NIC team in a host that is nothing to do with a virtual switch.  However, if I am using the NIC team for an external virtual switch, and I have fewer virtual NICs connecting to the virtual switch than I have team members, then I might use Address Hashing instead of Hyper-V Switch Port.

EDIT:

WS2012 R2 added a new load distribution mode called Dynamic. It is enabled by default and should be used. It is a blend of Address Hashing for outbound traffic and Hyper-V Port for inbound traffic. Microsoft urges you to use this default load balancing method on WS2012 R2.

END EDIT

This information has been brought to you by Windows Server 2012 Hyper-V Installation and Configuration Guide (available on pre-order on Amazon) where you’ll find lots of PowerShell like in this script:

image

 

Windows Server 2012 NIC Teaming Part 5 – Configuration Matrix

Technorati Tags: Windows Server 2012,Hyper-V,Networking

15 comments so far

Add Your Comment
  1. Hi Aidan , Thanks for these helpful articles,
    Can you tell me in windows server 2012 hyper-v how to map a Physical NIC(pnic) to a virtual NIC(vnic) , is there any way or powershell commands to find this?

    • Virtual Switch.

  2. Hi and many thanks to your great work and sorry for my bad English!

    I have a question regrading Hyper-V Port mode while configuring NIC Teaming. As you have said in your book and blog, by using this mode, the NIC Teaming feature specifies a single physical NIC for each vNICs in VMs. Right? But the following statement has confused me:

    Hyper-V Switch Port is appropriate if the team is being used for an external virtual switch (like the above examples) and:

    You have more virtual NICs than you have physical NICs. Maybe you have 2 physical NICs and 20 virtual machines. Maybe you have 2 physical NICs and you are creating a converged fabric design with 4 virtual NICs in the management OS and several virtual machines.

    If each NIC is configured to use a specific physical NIC, what do you mean regrading using this mode in situation while we have several vNICs with much lower physical NICs? For example, We have 20 VMs with 20 vNICs and a team with only two physical members with the speed of 1 GbE, this 20 vNICs share the 2 physical NICs’ bandwith, each vNICs have 100 MB bandwidth?

    Thanks

    • In your example, 10 vNICs will be round-robined to go through pNIC A and share its bandwidth, and the same will happen to the other 10 vNICs on pNIC B.

      No they won’t have 100 MB bandwidth. They will share 1 GbE. We virtualise because we are underconsuming physical hardware. We run sizing/assessment to determine how much physical resources are actually required by the VMs. I recommend you read chapter 1.

  3. Hi Aidan ,
    Have you experimented these addressing modes? Does it really works as you said? i am asking because i am experimenting them , i am using a Dell R610 server and Dell Powerconnect-8024 switch , i am testing OLTP performance through benchmark factory using windows server 2012 , and juding the packet distribution among three broadcom nics which i teamed by Wireshark, In Address Hash method i am not getting any equal distribution , and in hyper-v mode only 1 nic is dominating.
    And another question is if hyper-v mapped a v-nic to a p-nic , so what is the difference between hyper-v and SRIOV m as in SRIOV we have a dedicated virtual function to a physical function. Please elaborate me, i need it.

    Thanks

    • Please re-read the text. There is no guarantee of equal load distribution. Traffic is directed to physical NICs based on destination information in the packets. If server A is sending data to one service on server B, then traffic will go via one physical NIC.

      • Aiden, first of all thanks for very good article. It is the best description I have found so far. The theory is important but real test is always needed to proof the theory.
        Based on my experience I have to agree with Saurabh.
        I have File Server based on Win 2012 with LACP teaming of four 10Gb NICs with address hashing load balancing. I’m doing stress test (scripts) writing and reading files from 10 different workstations. IP addresses of workstations are in in sequence like 192.168.10.1,2,3,4,5,6,7,8,9,10. I can see well balanced traffic to file server (balanced by datacenter switch Force10 S4810) but traffic from file server to workstations is very unbalanced. Just my $0.02. David.

  4. Hi Aidan,

    I have a NIC team (LACP/Hyper-V Port) with 2 10GbE physical NICs. Any reason why when I build out a converged network with 3 vNICs I am getting an average throughput of about 5 Gb/s when I should be getting closer to 10 Gb/s?

    Thanks!

    • The answer is in the series of posts.

  5. hi aidan,

    reading mitch tulloch’s preview of his book “Introducing Windows Server 2012 R2″ i stumbled across the following in chapter#4 which is hardly to believe:

    „As a result, this mode should only be selected when the virtual machines running on the host have multiple virtual network adapters. If you use this mode when a virtual machine has only one virtual network adapter and the physical NIC being used by that adapter fails, that virtual machine will lose all connectivity with the network even though the host machine still has connectivity because of the fault tolerance of the NIC team.“

    what’s your take on this?

    thx

    • What is “this mode”? Hard to comment on an out-of-context quote.

  6. Hi Aidan,
    I’m setting up a 6 node Windows 2012 Hyper-V Cluster. Each host has 2 10Gb NICs ONLY that are teamed. The team needs to therefore carry Management, VM, Live Migration traffic. The vSwitches that have been configured on each host therefore have the “share with parent partition”. Given my scenario, should a use the Hyper-V port mode or the address hash mode on each of the teams?

    • See above.

  7. Hi Aidan,
    I have a two node Windows Server 2012 R2 failover cluster setup (diagram at the URI). Each node has a dual port 1 Gbps NIC and dual port 10 Gbps NIC. The 1 Gbps ports on each node are in a switch independent team using Dynamic distribution. The 10 Gbps ports on each node are in a team using the same configuration as the 1 Gbps ports. The cluster will host 4 VMs. I see above that you would recommend Hyper-V Port distribution in my scenario if I were using Windows Server 2012.

    Do you recommend Hyper-V Port distribution instead of Dynamic distribution in my scenario (Windows Server 2012 R2)?
    Also since the included link to my diagram shows a loop, will my switches need features to prevent loops such as the Spanning-Tree protocol?

    • The recommendation (which I agree with) from Microsoft is always use Dynamic load distribution in WS2012 R2. It gives you the best of Hyper-V Port and Address Hashing in one solution.

Get Adobe Flash player