NUMA has been one of those things that’s been with us for some time and gone unmentioned by most of us who didn’t work in the “super computing” high end of the market. But for us Hyper-V folks, it came to the fore when we got Dynamic Memory and needed to understand the penalties of a VM expanding it’s RAM across NUMA boundaries. It wasn’t a common possibility thanks to the limit of 4 vCPUs and 64 GB vRAM per VM, but it was a risk nonetheless.
Windows Server 2012 supports 1 TB RAM and 64 vCPUS per VM. On that host, we’ll definitely span NUMA nodes with that spec of VM. How does Windows Server 2012 Hyper-V react? Jeffrey Snover has the answer on the Windows Server 8 blog. Long story short: Hyper-V does the work for you, and hides the complexity unless you want to go looking for it.
The cleverness is that Hyper-V makes the guest OS aware of NUMA in the host. Windows VMs can then schedule their internal processes and memory according to the NUMA boundaries of that VM, just like a physical installation would have.
Using the ACPI SRAT for presenting NUMA topology is an industry standard, which means Linux and other operating systems that are NUMA aware can take advantage of Hyper-V virtual NUMA
While Hyper-V will figure out NUMA node sizes on the host automatically, what happens when you Live Migrate a VM to a different spec host? Well, the settings presented to the VM obviously cannot be changed while the VM is running. It’ll take the original NUMA node sizes to the new host and use them there. Tucked away in the advanced vCPU settings, you can customize a VM’s NUMA nodes to suit the host with the smallest NUMA nodes. That means the VM won’t span NUMA boundaries (aka Remote Run where it a process on a CPU is allocated RAM in a remote NUMA node) when it Live Migrates between hosts of different specs.
In a second post, Jeffrey Snover talks about these advanced settings, how to reset them back to the defaults, and how to monitor NUMA using Performance Monitor. In my opinion, these two posts are essential reading if you intend to do scale out computing on a virtualisation platform.
Now that big SQL, IIS 8.0, or MySQL workload (assuming MySQL is NUMA aware like SQL is since 2005) can be moved onto Hyper-V and take full advantage of the benefits of virtualisation and private cloud, without compromising on scale up performance demands.