2014
07.28

If you’re affected by this issue then you should have read this post. Microsoft posted a KB article for when virtual machines lose network connectivity when you use Broadcom NetXtreme 1-gigabit network adapters on Windows Server 2012 Hyper-V or Windows Server 2012 R2 Hyper-V.

Symptoms

When you have Hyper-V running on Microsoft Windows Server 2012 or Windows Server 2012 R2 together with Broadcom NetXtreme 1-gigabit network adapters (but not NetXtreme II network adapters), you may notice one or more of the following symptoms:

  • Virtual machines may randomly lose network connectivity. The network adapter seems to be working in the virtual machine. However, you cannot ping or access network resources from the virtual machine. Restarting the virtual machine does not resolve the issue.
  • You cannot ping or connect to a virtual machine from a remote computer.

These symptoms may occur on some or all virtual machines on the server that is running Hyper-V. Restarting the server immediately resolves network connectivity to all the virtual machines.

Cause

This is a known issue with Broadcom NetXtreme 1-gigabit network adapters that use the b57nd60a.sys driver when VMQ is enabled on the network adapter. (By default, VMQ is enabled.)

The latest versions of the driver are 16.2 and 16.4, depending on which OEM version that you are using or whether you are using the Broadcom driver version. Broadcom designates these driver versions as 57xx-based chipsets. They include 5714, 5715, 5717, 5718, 5719, 5720, 5721, 5722, 5723, and 5780.

These drivers are also sold under different model numbers by some server OEMs. HP sells these drivers under model numbers NC1xx, NC3xx, and NC7xx.

Workaround

Broadcom is aware of this issue and will release a driver update to resolve the issue. In the meantime, you can work around the issue by disabling VMQ on each affected Broadcom network adapter by using the Set-NetAdapterVmq Windows PowerShell command. For example, if you have a dual-port network adapter, and if the ports are named NIC 1 and NIC 2 in Windows, you would disable VMQ on each adapter by using the following commands:

Set-NetAdapterVmq -Name “NIC 1″ -Enabled $False
Set-NetAdapterVmq -Name “NIC 2″ -Enabled $False

You can confirm that VMQ is disabled on the correct network adapters by using the Get-NetAdapterVmq Windows PowerShell command.

Note By default, VMQ is disabled on the Hyper-V virtual switch for virtual machines that are using 1-gigabit network adapters. VMQ is enabled on a Hyper-V virtual switch only when the system is using 10-gigabit or faster network adapters. This means that by disabling VMQ on the Broadcom network adapter, you are not losing network performance or any other benefits because this is the default. However, you have to work around the driver issue.

Get-NetAdapterVmqQueue shows the virtual machine queues (VMQs) that are allocated on network adapters. You will not see any virtual machine queues that are allocated to 1-gigabit network adapters by default.

Sigh. I hope Broadcom are quicker about releasing a fix than Emulex (customers are waiting 10 or 11 months now?).

5 comments so far

Add Your Comment
  1. Hi Aidan, I have a 3 node Hyper-V cluster up on WS2012 with only Intel 1 gigabit adapters. Here’s Get-NetAdapterVmq output from one of the nodes – http://1drv.ms/1kiEMuG and VMQ is enabled. Fortunately I’m not having any issues described above with Broadcom adatpers, but I’m curious why is that :-)

    Dusan

    • That’s the nature of these things. Lots of variables and failure is not guaranteed.

  2. Broadcom Network adapters have historically been hit or miss. At my work, ESXi boxes have had issues with them, we have had Windows 2008 R2 SQL/Exchange servers that have issues and all those issues would clear up when replaced with Intel/Cisco cards.

    We finally got to point where we just order Intel cards in all new Dell boxes and call it a day.

  3. FYI We’re experiencing similar symptoms but with 10Gb Broadcom adapters and some of our VMs. It seems that older OS i.e. 2008 R2 and prior are affected. Restarting the guest OS doesn’t resolve anything but switching the network adapter to DHCP then set back again its static IP resolves the issue for us for some odd reason. Looks like something wonky is happening in the integration services/drivers for the vNIC.

    We haven’t tried disabling VMQ yet on the physical adapters. Maybe something to explore.

  4. That would explain a lot… I had that issue a few times with my NICs integrated in Dell R420 :/ I’ve had them in a team in WS2012R2 cluster node and I would loose all network connectivity to all VMs and the node. Though I did not need to reboot the server – in my case I had NICs in team as active-backup and I just needed to change active NIC to restore connectivity (logged in with idrac :) ). I see it’s a good thing we switched to Intel NICs.

    btw. I had another NIC in server for cluster only network and I wondered why that was working…

Get Adobe Flash player