November, 2009 | Aidan Finn, IT Pro

W2008 R2 Hyper-V Memory More Efficient

I’m seeing the real world results of this. We’re getting a little bit more out of the gigabytes of RAM that is in each of our hosts with Windows Server 2008 R2 Hyper-V instead of its predecessor, even with our hardware which does not have the very latest processor.

One of the main players in saving RAM on Hyper-V hosts with newer hardware will be SLAT or Second Level Address Translation.

In Windows Server 2008 Hyper-V the parent partition (host operating system) is responsible for mapping the physical memory in the host with the memory that is running in the virtual machine. Windows Server 2008 R2 Hyper-V removes that middle layer of management by offloading the responsibility to dedicated functions in the CPU.

CPU’s that have Intel’s Extended Page Tables (EPT) or AMD’s Nested Page Tables (NPT) or (Rapid Virtualization Indexing (RVI) have the ability to by delegated this responsibility. This gets rid of the “shadow table” that the parent partition otherwise had to use … which was also consuming RAM.

It’s estimated that you will save 1MB of RAM per VM (there is an overhead of RAM for every VM and GB RAM in that VM) and there is also a small saving in CPU time required.

Extending a W2008 R2 Hyper-V Cluster Shared Volume

Unlike VMware’s VMFS, we can extend a Cluster Shared Volume (CSV) without doing trickery that compromises performance. And has has been documented by Hans Vredevoort (clustering MVP) it is very scalable.

How do you resize or expand a CSV? It’s a pretty simple process:

Use your storage management solution (I’m using HP EVA Command View for the EVA SAN) to expand the size of the LUN or disk.
Use the Failover Clustering MMC to identify who is the CSV coordinator, i.e. the owner of the disk.
Log into the CSV coordinator.
Use either Computer Management->Storage Management or DISKPART.
Remember to rescan the disks.
Extend the volume to use all available space.

The steps for using DISKPART are:

rescan
list volume get the ID number for the CSV from here
select volume <ID Number> using the ID number from the previous step>
extend
list volume to see the results

It’s a painless operation and has no impact on running VM’s.

Virtualisation: The Undersold Truth

Ease of administration.

To a sys admin, those 3 words mean a lot. To a decision maker like a CIO or a CFO (often one and the same) they mean nothing.

It’s rare enough that I find myself working with physical boxes these days. Most everyone is looking for a virtualised service which is cool with me. Over the last 2 weeks I’ve been doing some physical server builds with Windows Server 2008 R2. I know the techniques for a automated installation. I just haven’t had time to deploy them for the few builds I needed to do. Things like Offline Servicing for VM’s and MDT/WDS (upgrade) are in my plans but things had to be prioritised. I’ve just kicked off a reboot of a blade server. By the time that’s finished it’s POST I’ll have made and half drunk a cup of coffee. After working with VM’s almost exclusively for the last 18 months, working with a physical box seems slow. These are fine machines but the setup time required seems slow. Those reboots take forever! VM reboots: well there’s no POST and they reboot extremely quickly.

Let’s compare the process of deploying a VM and a physical box

Deploy a VM

Deploy a VM.
Log in and tweak.
Handover the VM.

Notes on this:

The free Offline Servicing Tool can allow you to deploy VM’s that already have all the security updates.
This process can be done by a delegate “end user” using the VMM self servicing web interface.
The process was probably just an hour or two from end to end.

Deploy a Physical Server

Create a purchase request for a new server.
Wait 1-7 days for a PO number.
Order the server.
Wait for up to 7 days for the server to be delivered.
Rack, power and network the server.
We’ll assume you have all your ducks in a row here: Use MDT 2010 or ConfigMgr to deploy an operating system.
The OS installs and the task sequence deploys updates (reboots), then applications (reboots), then more updates (reboots) and then makes tweaks (more updates and a reboot).
You have over the server.

Notes on this:

Most people don’t automate a server build. Manual installs typically take 1 to 1.5 days.
There will probably be up to 1 day of a delay for networking.
The “end user” can’t do self service and must wait for IT, often getting frustrated.
The entire process will probably take 10.5 to 16.5 days.

Total Hardware Breakdown

Let’s assume the VM scenario used a cluster. If the hardware failure crashed the host then the VM stops running. The cluster moves the VM resource to another host (VMM will choose the most suitable one) and the VM starts up again. Every VM on the cluster has hardware fault tolerance. If the hardware failure was non-critical then you can use Live Migration to move all the VM’s to another host (VMM 2008 R2 maintenance mode) and then power down the host to work on it. There’s no manual intervention at all in keeping things running.

What if you used standalone (un-clustered) hosts. As long as you have an identical server chassis available you can swap the disks and network cables to get back up and running in a matter of minutes.

Unbelievably worst case scenario with un-clustered hosts: you can take the data disks and slap them into another machine and do some manual work to get running again. As long as the processor is from the same manufacturer you’re good to go in a few hours.

If a physical box dies then you can do something similar to that. However, physical boxes tend to vary quite a lot. A farm of virtualisation hosts don’t usually vary too much at all. If a DL380 dies then you can expect to put the disks into a DL160 and have a good result. It might work.

Most companies don’t purchase the “within 4 hours” response contracts. And even if they do, some manufacturers will do their very best to avoid sending anyone out by asking for one diagnostic test after another and endless collections of logs. It could be 1 to 3 days (and some angry phone calls) before an engineer comes out to fix the server. In that time the hosted application has been offline, negatively affecting the business and potentially your customers. If only a physical server was a portable container like a VM – see boot from VHD.

Summary

You’ve heard all those sales lines on virtualisation: carbon footprint, reduced rack space, lower power bills, etc. Now you can see how easier administration can make your life easier but positively impact the business.

My experience has been that when you translate techie-speak into Euros, Dollars, Pounds, Rubles, Yen or Yuan then that get’s the budget owners attention. The CFO will sit up and listen and probably decide in your favour. And if you can explain how these technologies will have real positive impacts on the business then the other decision makers will also have your attention.

Finished Our W2008 R2 Hyper-V Cluster Migration

Last night we finished migrating the last of the virtual machines from our Windows Server 2008 Hyper-V cluster to the new Windows Server 2008 R2 Hyper-V cluster. As before, all the work was done using System Center Virtual Machine Manager (VMM) 2008 R2. The remaining host has been rebuilt and is half way to being a new member of the R2 Hyper-V cluster.

I also learned something new today. There’s no supported way to remove a cluster from OpsMgr 2007. Yuk!

Happy Thanksgiving!

Happy Turducken day to our American friends. You’re probably not reading this until at least next Monday but you’ll at least know I was with you in spirit. I’m taking the day off as a Niners fan and cheering on the Lions and whomever is playing against the Cowboys.

Of course, being a Niners fan I am not doing the Turkey thing. I’ve got some Lasagne and a fine bottle of wine 🙂

W2008 R2 Hyper-V Network Tests on HP G6 Blades

I posted earlier today about my network transfer tests on HP ProLiant BL460C G5 blade servers with Windows Server 2008 R2 Hyper-V. Hans Vredevoort also did some tests, this time using BL460C G6 blades. This gave Hans the hardware to take advantage of some of the new technologies from Microsoft. Check out his results.

W2008 R2 Hyper-V Networking Enhancements

Windows Server 2008 R2 includes some enhancements to optimise how networking works in Hyper-V. I’m going to have a look at some of these now.

Virtual Machine Queue

Here’s the way things worked in Windows Server 2008. The NIC (bottom left) runs at the hardware level. VM1 has a virtual NIC.

When it communicates memory is copied to/from that NIC by the parent partition. All routing and filtering and data copying is done by the parent partition in Windows Server 2008.

Windows Server 2008 R2 takes advantage of Microsoft partnering with hardware manufacturers.

How it works now is that the NIC, i.e. the hardware, handles the workload on behalf of the parent partition. Hardware performs more efficiently than software. All that routing, filtering and data copy is handled by the network card in the physical host. This does rely on hardware that’s capable of doing this.

The results:

Performance is better overall. The CPU of the host is less involved and more available. Data transfer is more efficient.
Live Migration can work with full TCP offload.
Anyone using 10GB/E will notice huge improvements.

Jumbo Frames

TCP is pretty chatty. Data is broken up and converted into packets that must be acknowledged by the recipient. There’s an over head to this with the data being encapsulated with flow, control and routing information. It would be more efficient if we could send fewer packets that contained more data, therefore with less encapsulation data being sent.

Jumbo Packets accomplish this. Microsoft claims that you can get packets that contain 6 times more information with this turned on. It will speed up large file transfers as well as reduce CPU utilisation.

Chimney Offload

This one has been around for a while with Windows but is support for Hyper-V was added with Windows Server 2008 R2.

It’s similar to VMQ, requiring hardware support, and does a similar job. The NIC is more involved in doing the work. Instead of offloading from the parent partition, it’s offloading from the Virtual Machine’s virtual NIC. The virtual NIC in the VM advertises connection offload capabilities. The virtual switch in the parent partition offloads child partition TCP connections to the NIC.

Hardware Reliance

You need support from the hardware for these features. During the RC release, the following NIC’s were included by MS in the media:

VM-Chimney Capable Drivers:

Broadcom Net-Xtreme II 1 Gb/s NICs (Models 5706, 5708, and 5709)
Broadcom 10Gb/s NICs (Models 57710, 57711)

VMQ Capable Drivers:

Intel Kawela (E1Q) 1 Gb/s NICs (also known as Pro/1000 ET NICs)
Intel Oplin NICs (IXE) 10Gb/s NICs (also known as 82598)

W2008 R2 Hyper-V Network Speed Comparisons

Hans Vredevoort asked what sort of network speed comparisons I was getting with Windows Server 2008 R2 Hyper-V. With W2008 R2 Hyper-V you get new features like Jumbo Frames and VMQ (Virtual Machine Queue) but these are reliant on hardware support. Hans is running HP G6 ProLiant servers so he has that support. Our current hardware are HP G5 ProLiant servers. I decided this was worth a test.

I set up a test on our production systems. It’s not a perfect test lab because there are VM’s doing their normal workload and thing like continuous backup agents running. This means other factors that are beyond my control have played their part in the test.

The hardware was a pair of HP BL460C “G5” blades in a C7000 enclosure with Ethernet Virtual Connects. The operating system was Windows Server 2008 R2. The 2 virtual machines were also running Windows Server 2008 R2. I set them up with just 512MB RAM and a single virtual CPU. Both VM’s had 1 virtual NIC, both in the same VLAN. They had dynamic VHD’s. The test task would be to copy the W2008 R2 ISO file from one machine to the other. The file is 2.79 GB (2,996,488 bytes) in size.

There were three tests. In each one I would copy the file 3 times to get an average time required.

Scenario 1: Virtual to Virtual on the Same Host

I copied the ISO from VM1 to VM2 while both VM’s were running on host one. After I ran this test I realised something. The first iteration took slightly longer than all other tests. The reason was simple enough – the dynamic VHD probably had to expand a bit. I took this into account and reran the test.

With this test the data stream would never reach the physical Ethernet. All data would stay within the physical host. Traffic would route via the NIC in VM1 to the virtual switch via its VMBus and then back to the NIC in VM2 via its VMBus.

The times (seconds) taken were 51, 55 and 50 with an average of 52 seconds.

Scenario 2: Virtual to Virtual on Different Hosts

I used live migration to move VM2 to a second physical host in the cluster. This means that data from VM1 would leave the virtual NIC in VM1, traverse VMBus and the Virtual Switch and physical NIC in host 1, the Ethernet (HP C7000 backplane/Virtual Connects) and then the physical NIC and virtual switch in physical host 2 to reach the virtual NIC of VM2 via its VMBus.

I repeated the tests. The times (seconds) taken were 52, 54 and 66 with an average of 57.333 seconds. We appear to have added 5.333 seconds to the operation by introducing physical hardware transitions.

Scenario 3: Virtual to Virtual During Live Migration

With this test we would start with the scenario in the first set of tests. We would introduce Live Migration to move VM2 from physical host 1 to physical host 2 during the copy. This is why I used on 512MB RAM in the VMs; I wanted to be sure the live migration end-to-end task would complete during the file copy. The resulting scenario would have VM2 on physical host 2, matching the second test scenario. I want to see what impact Live Migration would have on getting from scenario 1 to scenario 2.

The times (seconds) taken were 59, 59 and 61 with an average of 59.666 seconds. This is 7 seconds slower than scenario 1 and 2.333 seconds slower than scenario 2.

Note that Live Migration is routed via a different physical NIC than the virtual switch.

Scenario 4: Physical to Physical

This time I would copy the ISO file from one parent partition to another, i.e. from host 1 to host 2 via the parent partition NIC. This removes the virtual NIC, virtual switch and the VMBus from the equation.

The times (seconds) taken were 34, 28 and 27 with an average of 29.666 seconds. This makes the test scenario physical data transfer 22.334 seconds faster than the fastest of the virtual scenarios (scenario 1).

Comparison

Scenario	Average Time Required (seconds)
Virtual to Virtual on Same Host	52
Virtual to Virtual on Different Hosts	57.333
Virtual to Virtual During Live Migration	59.666
Physical to Physical	29.666

Waiver

As I mentioned, these tests were not done in lab conditions. The parent partition NIC’s had no traffic to deal with other than an OpsMgr agent. The Virtual Switch NIC’s had to deal with application, continuous backup, AV and OpsMgr agent traffic.

It should also be noted that this should not be a comment on the new features Windows Server 2008 R2 Hyper-V. Using HP G5 hardware I cannot avail of the new hardware offloading improvements such as VMQ and Jumbo Frames. I guess I have to wait until our next host purchase to see some of that in play!

This is just a test of how things compare on the hardware that I have in a production situation. I’m actually pretty happy with it and I’ll be happier when we can add some G6 hardware.

Lots Of Operations Manager Updates

Microsoft released lots of updates for Operations Manager over the last couple of weeks. There are lots of updates to management packs, too many for me to go posting them at this time of night. Have a look on the catalogue and you’ll see them. Or check your console if you’re using OpsMgr 2007 R2.

Most importantly is KB971541, Update Rollup for Operations Manager 2007 Service Pack 1.

“The Update Rollup for Operations Manager 2007 Service Pack 1 (SP1) combines previous hotfix releases for SP1 with additional fixes and support of SP1 roles on Windows 7 and Windows Server 2008 R2. This update also provides database role and SQL Server Reporting Services upgrade support from SQL Server 2005 to SQL Server 2008.

The Update Rollup includes updates for the following Operations Manager Roles:

Root Management Server, Management Server, Gateway Server
Operations Console
Operations Management Web Console Server
Agent
Audit Collection Server (ACS Server)
Reporting Server

The following tools and updates are provided within this update which may be specific to a scenario:

Support Tools folder – Contains SRSUpgradeTool.exe and SRSUpgradeHelper.msi (Enables upgrade of a SQL Server 2005 Reporting Server used by Operations Manager Reporting to SQL Server 2008 Reporting Server)
Gateway folder – Contains a MSI transform and script to update MOMGateway.MSI for successful installation on Windows Server 2008 R2
ManagementPacks folder – Contains an updated Microsoft.SystemCenter.DataWarehouse.mp which requires manual import

For a list of fixes and tools addressed by this update rollup, see KB971541.

This update is supported for application on System Center Operations Manager 2007 Service Pack 1 only.

Feature Summary

The System Center Operations Manager 2007 SP1 Rollup 1 contains:

All binary hotfixes released since Service Pack 1 release
Support for Windows 7 and Windows Server 2008 R2
Operational and DataWarehouse database support on Windows Server 2008 R2
Additional stability hotfixes”

Requirements

Supported Operating Systems: Windows 7; Windows Server 2003; Windows Server 2008; Windows Server 2008 R2; Windows Vista; Windows XP
System Center Operations Manager 2007 Service Pack 1

Instructions

This update must be applied to each computer that meets the following criteria:

Hosts a Microsoft Operations Manager Root Management Server
Hosts a Microsoft Operations Manager Management Server
Hosts a Microsoft Operations Manager Operations Console
Hosts a Microsoft Operations Manager Web Console Server
Hosts a Microsoft Operations Manager Reporting Server
Hosts a Microsoft Operations Manager Manually installed Agent
Hosts a Microsoft Operations Manager ACS Server

Before applying this update it is strongly recommended that Operations Manager databases, Management Server, Report Server and Web Console roles be backed up.

To extract the files contained in this update and installation of the update on the Operations Manager roles above:

Copy the file – SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI – To either a local folder or accessible network shared folder.
Run the file – SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI – locally on each applicable computer that meets the predefined criteria.
You can run SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI from either Windows Explorer or from a command prompt.
Select the appropriate role to update from the Operations Manager 2007 Software Update dialog.

NOTE: To run this file on Windows Server 2008 you must run this file from a command prompt which was executed with the Run as Administrator option. Failure to execute this Windows installer file under an elevated command prompt will not allow display of the System Center Operations Manager 2007 Software Update dialog to allow installation of the hotfix”.

Upgrading Active Directory Domains to Windows Server 2008 and Windows Server 2008 R2 AD DS Domains

This guide explains the process for upgrading Active Directory domains to Windows Server 2008 and Windows Server 2008 R2, how to upgrade the operating system of domain controllers, and how to add domain controllers that run Windows Server 2008 or Windows Server 2008 R2 to an existing domain.

Upgrading your network operating system requires minimal network configuration and typically has a low impact on user operations. The upgrade process is straightforward, efficient, and allows your organization to take advantage of the improved security that is offered by the Windows Server 2008 and Windows Server 2008 R2 operating systems. This guide covers the process for upgrading domains and domain controllers, and how to add new domain controllers to existing Active Directory domains. It includes details about how to run Adprep.exe and resolve known issues and errors if they arise.