Numerous sources have reported that KB3161606, an update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (WS2012 R2), are breaking the upgrade of Hyper-V VM integration components. This has been confirmed & Microsoft is aware of the situation.
As noted below by the many comments, Microsoft eventually released a superseding update to resolve these issues.
The scenario is:
- You deploy the update to your hosts – which upgrades the ISO for the Hyper-V ICs
- You deploy the update to your VMs because it contains many Windows updates, not just the ICs.
- You attempt to upgrade the ICs in your VMs to stay current. The upgrade will fail.
Note that if you upgrade the ICs before deploying the update rollup inside of the VM, then the upgrade works.
My advice is the same as it has been for a while now. If you have the means to manage updates, then do not approve them for 2 months (I used to say 1 month, but System Center Service Manager decided to cause havoc a little while ago). Let someone else be the tester that gets burned and fired.
Here’s hoping that Microsoft re-releases the update in a way that doesn’t require uninstalls. Those who have done the deployment already in their VMs won’t want another painful maintenance window that requires uninstall-reboot-install-reboot across all of their VMs.
EDIT (6/7/2016)
Microsoft is working on a fix for the Hyper-V IC issue. After multiple reports of issues on scale-out file servers (SOFS), it’s become clear that you should not install KB3161606 on SOFS clusters either.
Why can’t Microsoft revoke updates with issues, or at least post this on the KB?
Great info; thanks! Do you know if the WS2012 version, KB3161609, which also includes the same IC update, has the same issue?
Didn’t hear anything.
This update also broke my SOFS cluster which is serving backing virtual disk files to a Hyper-V cluster. Caused access to the fileshares hosted on the SOFS node where this was applied to freeze and reboot those VMs.
Yours was the first of 2 reports I’ve heard via this blog. I’ve contacted PMs in Microsoft to let them know.
Any news on Microsoft side? They have yet to acknowledge this issue (it happened to my systems as well). Should we wait on them or just remove the whole KB3161606 ?
Wait – they are working on a fix.
This update broke two of our Storage Spaces Hyper-Converged Cluster Systems. We uninstalled KB3161606. After that everything went fine again.
That’s the second report I’ve heard via the blog. I’ve contacted PMs in Microsoft to let them know.
We’ve had issues on our SOFS, specifically with HV and SQL Clusters losing access to the Quorum Shares hosted on the SOFS. Subsequently, the clusters lost their config information and services crashed (VMs and SQL instances).
We made the mistake of installing this onto our Hyper-V cluster today. Massive issues with live migration, VMs getting locked, vmwp.exe getting locked, vmms.exe getting locked, VMs disconnecting from the network, the RHS deadlocking and crashing, and cluster nodes refusing to shutdown.
Has MS fixed this issue? What is the current best workaround, so we can the benefit of the other hot-fixes in this rollup?
I haven’t heard anything about a fix 🙁
The workaround is messy. Take ownership of winsxs\Manifests. Then rename the manifest to *.old and install the updated integration components. I’d wait for an update from Microsoft.
And why isn’t there known issues documented anywhere in the KB?
That’s not unusual 🙁
Thanks, Aidan. I’ve found that there are no constraints on live migration after installing the rollup on Hyper-V hosts and guests, as long as I don’t try to install the updated IC’s. Hyper-V Mangler whines mightily, but that seems to be the extent. What are other folks seeing? (We don’t have any production SOFS.)
Following up on my prior question, the June rollup for WS2012, KB3161609 does, in fact, have the same issue. KB31361609 was re-released 12 Jul, and the KB article reflects that the revised update fixes problems installing the new IS on guests.
I have installed this update at about half the schools I look after before trying to run the integration services update, at which point I stopped.
At the schools I did install the update should I un-install or wait for MS to release some info.
At the schools I have not installed the update should I install or wait, as it has other fixes in the updated.
Thanks for your help
Wait for information from Microsoft
The July 2016 rollup has been released, KB3172614:
https://support.microsoft.com/en-us/kb/3172614
MS do not explicitly state that there is a fix included for the IC’s issue in the June rollup, but the notes state that this rollup replaces both May and June rollups. Have you heard anything from the MS folks about this issue as this morning?
TIA
Please be aware that the June update has now been rolled up into https://support.microsoft.com/en-us/kb/3172614 The July update rollup. I can’t tell if the issue has been fixed in the July release. Can you confirm with your sources?
This update rollup is superseded by July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) that was released on July 21, 2016.
Microsoft just released an update to KB3161606 on 22-Jul-2016 that corrects the issue. I applied the update to the host, rebooted, then I was able to successfully upgrade the IC on the hosts by inserting the virtual update DVD from the console without first uninstalling the prior update or the IC services. I then applied the update to the VM guests for good measure. Unfortunately this required a reboot of the host and 2 reboots of each VM. Very messy.
Microsoft has released KB3172614 for replace KB3161606.
https://support.microsoft.com/en-us/kb/3161606
https://support.microsoft.com/en-us/kb/3161606 says the June WS2012R2 Rollup has been replaced by the July WS2012R2 Rollup, released 21 Jul 2016, which includes fixes for the issues discussed in this blog and its comments. The July WS2012R2 Rollup is https://support.microsoft.com/en-us/kb/3172614.
AH HA:
Just updated the July KB3161606:
Known issues in this update
•Issue 1
Symptoms
Running data deduplication on a Cluster Shared Volume (CSV) on Windows Server 2012 R2 based-Hyper V cluster may impact live migration where resource hosting subsystem (RHS) may deadlock and crash with 0x9E Stop error.
Cause
This issue occurs because of an incorrect status returned by data deduplication driver to the CSV file system causing a 0x9E Stop error.
Resolution
To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).
•Issue 2
Symptoms
After this update is installed, you may receive an error message in certain configurations on a Hyper-V guest when you install Integration Services. The error message resembles the following message:
An error has occurred: One of the updated processes returned error code 14101.
Resolution
To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) on the Hyper-V host.
•Issue 3
Symptoms
After you install this update on a Hyper-V host, you may be unable to install this update in certain configurations on a Hyper-V guest if Integration Services have been updated on the guest.
Resolution
To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).
It looks like KB3161606 has now been superseded by KB3172614. I just wonder how safe it is 🙂 https://support.microsoft.com/en-us/kb/3172614
Looks like the kb3172614
(https://support.microsoft.com/en-us/kb/3172614)
appears to be the updated version that incorporates KB3161606. I did not see the issue in a test environment I saw with KB3161606. I will delay deployment to a production environment for a month or so if no issues crop up.
“The July 2016 update rollup includes some new improvements and fixes including the improvements from June 2016 update rollup KB3161606 and May 2016 update rollup KB3156418 for the Windows RT 8.1, Windows 8.1, and 2012 R2 platform. We recommend that you apply this update rollup as part of your regular maintenance routines. Before you install this update, check out the Prerequisites and restart requirement section”
Update replacement information
This update replaces the previously released updates 3161606 and 3156418.”
should also add to my comment that the integration service is a newer version than that of KB3161606
6.3.9600.18339 = KB3161606
6.3.9600.18398 = KB3172614
Is there any publication that outlines what is new with the newer integration service updates
Looks like Microsoft superseded this update with a new one (KB3172614). It would be great if you could do a followup blog post on this issue.
This latest update rollup https://support.microsoft.com/en-us/kb/3172614 seems to fix this problems. I have two 2012 r2 Hyper-V hosts and both of them had huge problems after KB3161606. Lots of network issues with quest VM’s and not being able to update hyper-v integration components. After installing 3172614 to host I was able to update the VM’s integration components and for now I have not seeing any network issues.
It looks like Microsoft has released an update. Has anybody tried this?
https://support.microsoft.com/en-us/kb/3172614
This update has now disappeared from my WSUS
Hi, looks like Microsoft fixed it.
https://support.microsoft.com/en-us/kb/3172614
The KB article is updated and the July 2016 update rollup KB3172614 is supposed to fix the issues. Did any of you already verify if this update indeed fixes the issues and is safe to install?
Any news on this?
If there is, Microsoft hasn’t shared it with us yet.
It looks like KB3161606 caused the NFS to crash also. Whenever we copy a large file from Linux system to our server 2012 r2 it crash the server. I updated the new 3172614, no luck. It still crash. 🙁