Microsoft added a new feature in Windows Server 2012 R2 Hyper-V called Virtual RSS or vRSS. Receive Side Scaling (RSS) is a feature used in physical NICs to allow a server’s networking capacity to scale out. Microsoft describes RSS as:
… a network driver technology that enables the efficient distribution of network receive processing across multiple CPUs in multiprocessor systems.
Long story, short, without RSS inbound networking scalability in a server is bottlenecked by the processing power of Core 0 of CPU 0, no matter how many cores or processors that you have in that server; all the networking interrupts go to that single core. With RSS in a NIC and enabled in the advanced settings (changes depending on your NIC) then Windows Server 2012 and later can spread the processing load across the cores/processors in the server (including Hyper-V hosts).
As I said, RSS is a function of physical NICs, and therefore a function of physical servers/hosts. Since WS2012 Hyper-V we have the ability to create massive VMs with 64 virtual processors, 1 TB RAM, and oodles of VHDX storage. I bet some of those workloads would like to do lots of inbound networking. Unfortunately, there was no RSS that you could use in the Hyper-V Virtual Ethernet Adapter.
That changes with WS2012 R2 Hyper-V because it allows us to turn on vRSS. The concept is simple enough. In the host you have one or more 10 Gbps or faster physical NICs (pNICs) that are connected to the virtual switch, probably via a NIC team (if there is more than one pNIC). Virtual Machine Queue (VMQ) is enabled in this NICs. VMQ is a cousin of RSS; it uses the same circuitry to increase the scalability of inbound VM networking on the host.
You can check the status of VMQ using PowerShell if you want using Get-NetAdapterVMQ.
You then enable vRSS in the properties of the virtual NIC in the guest OS of the VM. I went one step further by enabling Jumbo Frames.
Alternatively you can enable vRSS in the VM by using PowerShell in the guest OS if you want:
Enable-NetAdapterRSS –Name <AdapterName>
Voila; you have increased the scalability of the VM’s networking … assuming that the VM has lots of virtual prcessors. Now you can push some data to your VM(s). Open up Task Manager and the logical processors view and you can see the workload being scaled out across the virtual processors of the VM.
My reason for enabling vRSS was that I was building a second Scale-Out File Server, which needed to be virtual due to equipment shortages. I built up 2 VMs with Shared VHDX. Each VM in the virtual SOFS has 3 virtual NICs:
- Management: connected to a 1 Gbps virtual switch
- Storage1: connected to a 10 Gbps virtual witch, with dVMQ enabled on the pNIC
- Storage2: connected to another 10 Gbps virtual witch, with dVMQ enabled on the pNIC
All my cluster and SMB 3.0 traffic goes over Storage1 and Storage2 via SMB Multichannel. I enabled vRSS in the guest OS of the VMs. The above CPU performance was snapped from when I had a job running to create 60 WS2012 R2 VMs on the virtual SOFS. You can see that the processor load is better balanced … better for the VM’s scalability and probably better for the VM’s neighbours on the host.
Note that the supported guest OS’s are Windows 8.1 and Windows Server 2012 R2.
Note: NVGRE is compatible with vRSS. SR-IOV is not compatible with vRSS
Hi Aidan,
You wrote: “Voila; you have increased the scalability of the VM’s networking … assuming that the VM has lots of virtual NICs.”
Did you actually mean “… assuming that the VM has lots of virtual CPUs.”
Yes.
Hi Aidan,
do you advise Broadcom or Intel NICs? In combination with Dell servers & HP Procurve switches.
thanks,
Haroun.
Short answer: yes.
great article yaar, thanks a lot
Does vRSS also work on the host vNIC? e.g live migration or Management vNIC?
Unfortunately, no.
Great. What about the vSwitch performance with RSS enabled on the host?
Does vSwitch run on the different CPU of the host?
You’ve got the concept wrong. It’s the queues inside the guest OS of the VMs that are optimized, not the virtual switch. There’s no point in optimizing the switch.
We are running file server on Windows Server 2012 R2 on vmware server 5.1. I have enabled RSS in the properties of the virtual NIC in the guest OS of the VM but still it’s showing one CPU instead of 4 CPU in task manager.
You’ve completely misunderstood RSS. vRSS takes advantage of vCPUs that are added to a VM; it doesn’t add vCPUs to a VM. And VMware? Go that that question somewhere else 🙂
I’ve searched around on many of your blog pages to see if you had a write up similar to this design:
http://blogs.technet.com/b/cedward/archive/2014/02/28/hyper-v-2012-r2-network-architectures-part-8-of-7-bonus.aspx
Do you ever recommend teaming the VM nics?
That Microsoft diagram is very unclear and I have no idea what the author is trying to share. I never implement NIC teaming in a VM unless it’s to demonstrate something. It’s pointless (one exception: SR-IOV which is rare to find in the real world) if the virtual switch is connected to a team of physical NICs.
Typo –
Hi Aidan,
Is RSS and VMQ enabled on the physical storage NICs of both your SOFS nodes and Hyper-V Cluster Nodes.
Also is there a good article on setting up this part of the deployment.
Thanks for sharing.
RSS is required for throughput so ensure that it is enabled.
Hi Aidan,
Is there a limit on number of queues per VM that can be created for vRSS?
THanks a lot for your article.
Question. I get confused when SRIOV and VMQ need to be disabled for RSS or vRSS to work. Could you clarify that?
No need to do anything to SR-IOV or VMQ for RSS to work on the host. VMQ is needed for vRSS to work in the VM. But remember that this is for 10 Gbps or faster. Turn off VMQ on 1 GbE because it will break your networking.
Do I need to set also maximum number of rss processors or just enabling RSS is sufficient?
That’s a big question 🙂
Yes, that’s something I’m interested in and I can not find the answer. If we leave it as default(no explicit set CPU number), whether RSS will automatically balance load to all CPU.
Yes, that’s something I’m interested in and I can not find the answer. If we leave it as default(no explicit set CPU number), whether RSS will automatically balance load to all CPU.