Please note the below comments.Â There are memory leak issues being reported with this hotfix.
Please pay special attention to this hotfix.Â Itâ€™s the sort of one I expect to see on forums and be asked about for the next 18 months.Â I recommend making this patch a standard part of your install of WS2012 Hyper-V clusters.
The scenario is when a virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster.
Consider the following scenario:
- You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.
- You create a virtual machine on a CSV volume on a cluster node.
- You start the virtual machine.
- You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM) or any backup software that uses the Microsoft Software Shadow Copy Provider.
In this scenario, one of the following issues occurs:
- The backup is created, and the virtual machine enters a paused state.
- The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.
Additionally, the following events are logged in the Cluster log and System log respectively:
Software snapshot creation on Cluster Shared Volume(s) (‘volume location‘) with snapshot set id ‘snapshot id‘ failed with error ‘HrError(0×80042308)(2147754760)’. Please check the state of the CSV resources and the system events of the resource owner nodes.
Log Name: System
Date: Date and time
Event ID: 5120
Task Category: Cluster Shared Volume
Computer: Computer name
Description: Cluster Shared Volume ‘Volume1′ (‘nameâ€™) is no longer available on this node because of ‘STATUS_IO_TIMEOUT(c00000b5)’. All I/O will temporarily be queued until a path to the volume is reestablished.
Log Name:Â Â Â Â Â System
Source:Â Â Â Â Â Â Â Microsoft-Windows-FailoverClustering
Date:Â Â Â Â Â Â Â Â Â Date and time
Event ID:Â Â Â Â Â 5142
Task Category: Cluster Shared Volume
Level:Â Â Â Â Â Â Â Â Error
User:Â Â Â Â Â Â Â Â Â SYSTEM
Computer:Â Â Â Â Â Computer name
Description: Cluster Shared Volume ‘Volume3′ (‘Cluster Disk 4′) is no longer accessible from this cluster node because of error ‘ERROR_TIMEOUT(1460)’. Please troubleshoot this node’s connectivity to the storage device and network connectivity.
The virtual machine enters a paused state because the Ntfs.sys driver incorrectly reports the available space on the CSV volume when the backup software tries to create a snapshot of the CSV volume. Additionally, the CSV volume goes offline because the CSV volume does not resume from a paused state after an I/O delay issue or an I/O error occurs.
Note The CSV volume is resilient.
A supported hotfix is available from Microsoft.
There is more:
After you install the hotfix, CSV volumes do not enter paused states as frequently. Additionally, a cluster’s ability to recover from expected paused states that occur when a CSV failover does not occur is improved.
To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.
In this situation, disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to 1. For more information about how to disable ODX, go to the Microsoft website.
This blog post is the property of Aidan Finn (@joe_elway / http://www.aidanfinn.com) and may not be reused in any manner without prior consent of Aidan Finn. You may quote one paragraph from this blog post if you link to the original blog post.
- Troublesome KB2799728 (CSV Paused State/Offline Patch) Replaced By KB2813630
- KB2823643 – VMs Freeze At "Stopping" State After You Shutting Them Down On WS2012 Hyper-V Cluster
- Windows Server Backup Supports WS2012 Hyper-V Clusters
- KB2838669–A Big Hotfix Bundle For WS2012 Failover Clustering
- KB2770917 – WS2012 Hyper-V Backup Fails On NetApp