Sometimes, IT issues are not what they seem to be. A strange issue reared its ugly head last week regarding something I hold dearly: Active Directory backups.

The situation

An organization runs Active Directory Domain Controllers virtually on top of VMware vSphere. The VMware Tools are installed on the virtual machine.

The organization creates backups of the virtual machines. Instead of using snapshots or the Volume Shadow Copy writers from Windows, the organization relies on VMware Tools Quiescence:

Quiescing is the process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level, application-specific tasks.

The issue

When restoring the Domain Controller from backup, data loss occurred. To be precise, a USN rollback occurred. Since Windows Server 2012, Active Directory Domain Services has virtualization safeguards to prevent USN rollbacks in cases like this.

The cause

VMware Tools’ Quiescence feature prevented the VM Generation ID feature from working correctly.

The solution

Make backups of Domain Controllers using the volume shadow copy writers. Veeam’s Application-Aware Processing allows for these kinds of backups.

After a backup, check the domain controller’s event logs through Event Viewer (eventvwr.exe). Event ID 1917 provides the confidence that your configuration is properly triggering the VSS writer.

However, some applications still take a consistent backup of Active Directory without generating this event, as long as they trigger the VSS writer. Ensure that your application is set to at least take a System State backup and use the VSS writer, then look in the Application log for several Event 2001 and Event 2003 entries generated by ESENT. If there are no associated errors, then your directory is being safely backed up.

Further reading

Active Directory Virtualization Safeguards with VM-GenerationID on VMware vSphere 
VMware: "To Quiesce or not to Quiesce?" 
Application-Aware Processing 
VMware Tools Quiescence