KB2799728–VM Enters Paused State Or CSV Goes Offline When Backup WS2012 Hyper-V Cluster

Please note the below comments.  There are memory leak issues being reported with this hotfix.

Please pay special attention to this hotfix.  It’s the sort of one I expect to see on forums and be asked about for the next 18 months.  I recommend making this patch a standard part of your install of WS2012 Hyper-V clusters.

The scenario is when a virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster.

Consider the following scenario:

• You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.
• You create a virtual machine on a CSV volume on a cluster node.
• You start the virtual machine.
• You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM) or any backup software that uses the Microsoft Software Shadow Copy Provider.

In this scenario, one of the following issues occurs:

• The backup is created, and the virtual machine enters a paused state.
• The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.

Additionally, the following events are logged in the Cluster log and System log respectively:

Software snapshot creation on Cluster Shared Volume(s) (‘volume location‘) with snapshot set id ‘snapshot id‘ failed with error ‘HrError(0x80042308)(2147754760)’. Please check the state of the CSV resources and the system events of the resource owner nodes.

 

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume ‘Volume1’ (‘name’) is no longer available on this node because of ‘STATUS_IO_TIMEOUT(c00000b5)’. All I/O will temporarily be queued until a path to the volume is reestablished.

 

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          Date and time
Event ID:      5142
Task Category: Cluster Shared Volume
Level:         Error
Keywords:
User:          SYSTEM
Computer:      Computer name
Description: Cluster Shared Volume ‘Volume3’ (‘Cluster Disk 4’) is no longer accessible from this cluster node because of error ‘ERROR_TIMEOUT(1460)’. Please troubleshoot this node’s connectivity to the storage device and network connectivity.

The virtual machine enters a paused state because the Ntfs.sys driver incorrectly reports the available space on the CSV volume when the backup software tries to create a snapshot of the CSV volume. Additionally, the CSV volume goes offline because the CSV volume does not resume from a paused state after an I/O delay issue or an I/O error occurs.
Note The CSV volume is resilient.

A supported hotfix is available from Microsoft.

There is more:

After you install the hotfix, CSV volumes do not enter paused states as frequently. Additionally, a cluster’s ability to recover from expected paused states that occur when a CSV failover does not occur is improved.

To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.

In this situation, disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to 1. For more information about how to disable ODX, go to the Microsoft website.