Troubleshooting stories from the field are the best. That’s why I like writing them down. Although, sometimes they might appear as straight cases of schadenfreude, I feel there are lessons to be learned for anyone, if you’re willing to look closely and listen carefully.
Today, I saw someone stress over an ‘Oops!’ situation that occurred straight after a long Exchange to Office 365 migration weekend, that, as a consequence, went sour all of a sudden. Thinking he was doing the right things, the person actually made it worse. But, it’s a long story, so let’s begin at the beginning.
After a long migration weekend, involving a Microsoft Exchange to Office365 migration and Azure AD Connect for Hybrid Identity, someone on the project team accidentally deleted an entire Organizational Unit (OU) in Active Directory Domain Services. By the time the error was detected, Azure AD Connect had synchronized the changes and orphaned the Office365 mailboxes for the deleted user accounts…
Although the Active Directory environment consisted of Windows Server 2012-based Domain Controllers running the Windows Server 2012 Domain Functional Level (DFL) and Forest Functional Level (FFL), it didn’t have the Active Directory Recycle Bin enabled.
Getting the objects back, then, would pose a challenge. Not for these guys, because they use Veeam Backup & Replication with the Veeam Explorer for Active Directory.
However, because restores might present new other challenges, the person next to me decided to enable the Active Directory Recycle Bin feature in the Active Directory environment, before restoring the objects.
About VEEAM Explorer for Microsoft Active Directory
Veeam Backup & Replication offers ground-breaking restore possibilities, based on a single (incremental) backup file. The Veeam Explorer for Microsoft Active Directory is a feature that allows for object-level restores in Active Directory, based on the same single backup of any Domain Controller.
To this purpose, Veeam Backup & Replication creates an isolated environment to which it restores the Domain Controller, plus a Veeam interface to restore objects to the production Active Directory Domain Services environment.
After verifying successful replication of enabling the Active Directory Recycle Bin, he started up Veeam Backup & Replication and restored the affected objects using the Veeam Explorer for Microsoft Active Directory.
To his loud dismay, Veeam created objects with new Security Identifiers (sIDs), instead of properly restoring objects with their original sIDs, as he expected from the documentation and previous experiences.
Looking up to an enormous amount of work to hard match the new user objects with the orphaned mailboxes, correcting profile issues, etc. the person asked me for advice.
Now, of course, I know the Veeam Explorer for Microsoft Active Directory is not as bad as the person would have me think.
How the Active Directory Recycle Bin works
Let’s look at what happens when we enable the Active Directory Recycle Bin, introduced with Windows Server 2008 R2, first.
In an environment that doesn’t have the Active Directory Recycle Bin enabled, an object that is deleted becomes tombstoned for the period of the tombstone lifetime. This makes sure Domain Controllers have the opportunity to replicate this change. As part of the tombstoning process, the object is stripped from all link-valued and non-link-valued attributes, so the object doesn’t appear as a group member, etc. After the Tombstone Lifetime Period, the locally running garbage collection process deletes the tombstone object from the database, on each Domain Controller separately.
After the Active Directory Recycle Bin feature is enabled, the deletion process for objects changes, significantly, as the above figure shows:
- A deleted lifetime is introduced
After you enable Active Directory Recycle Bin, when an object is deleted, the system preserves all of the object's link-valued and non-link-valued attributes, and the object becomes logically deleted, which is a new state. A deleted object is moved to the Deleted Objects container, and its distinguished name is mangled. A deleted object remains in the Deleted Objects container in a logically deleted state throughout the duration of the deleted object lifetime. Within the deleted object lifetime, you can recover a deleted object with Active Directory Recycle Bin and make it a live Active Directory object again. Within the deleted object lifetime, you can also recover a deleted object through an authoritative restore.
- The tombstone lifetime is replaced with a recycled lifetime
After the deleted object lifetime expires, the logically deleted object is turned into a recycled object and most of its attributes are stripped away. After the recycled object lifetime expires, the garbage-collection process physically deletes the recycled Active Directory object from the database.
These changes result in a couple of situations, as documented by Microsoft:
The process of enabling Active Directory Recycle Bin is irreversible. After you enable Active Directory Recycle Bin in your environment, you cannot disable it.
When Active Directory Recycle Bin is enabled, all objects that were deleted before Active Directory Recycle Bin was enabled (that is, all tombstone objects) become recycled objects. These objects are no longer visible in the Deleted Objects container, and they cannot be recovered with Active Directory Recycle Bin. The only way to restore these objects is though an authoritative restore from a backup of AD DS that was taken of the environment before Active Directory Recycle Bin was enabled.
A recycled object cannot be recovered with Active Directory Recycle Bin or with the steps in Reanimating Active Directory Tombstone Objects.
How the Veeam Explorer for Microsoft Active Directory works
Now, let’s look at how the Veeam Explorer for Microsoft Active Directory works, from a functional perspective. In Veeam’s support forums, we found the following gem of information:
In case you need to restore an object then Veeam looks for it either in a tombstone container or in a recycle bin (in your case that would be a recycle). If nothing's found then Veeam needs to restore the object from backup. If you restore the object from a backup then your GUID and SID will change. In case of a tombstone/resyscle restoration GUID and SID do not change.
So, Veeam explorer for Microsoft Active Directory looks for deleted objects (when Active Directory Recycle Bin is enabled) and tombstone objects (when Active Directory Recycle Bin is not enabled) and tries to undelete or reanimate these (resulting in objects with the same sIDs), before resorting to recreate objects (resulting in objects with new sIDs).
Because the Active Directory Recycle Bin was enabled before Veeam Explorer for Microsoft Active Directory could work its magic, Veeam Explorer for Microsoft Active Directory was able to only find recycled objects, that can’t be undeleted or reanimated. Thus, Veeam Explorer for Microsoft Active Directory recreated the objects.
Microsoft’s documentation already points to the solution. Authoritatively restoring the objects should bring them back with their original sIDs.
We first deleted the objects that we created using Veeam Explorer for Active Directory, to avoid duplicate objects. Then, again, Veeam Backup & Replication helped in this case. We performed an authoritative restore for the Organizational Unit (OU) from backup, using the following commands in Directory Services Restore Mode (DSRM):
activate instance ntds
restore subtree “OU=OrganizationalUnit,DC=domain, DC=tld
Can’t get into the Directory Services Restore Mode (DSRM), because the startup option is missing on your Domain Controller? Follow these instructions.
This resulted in objects with original sIDs, and no more orphaned mailboxes, profile issues, etc.
Under the hood, Active Directory works in mysterious ways. Understanding how it works, makes your life easier and your choices better.
Backup solutions know how Active Directory works, and allow for ways to handle possible restrictions. The people at Veeam have found excellent ways to handle Active Directory and I still feel the Veeam Explorer for Microsoft Active Directory is a tremendously intelligent way to handle Active Directory object restores, with authoritative restores as a worst case help line if you need it.
How to add a DSRM startup option in Windows Server 2008 and 2008 R2
I am a 2016 Veeam Vanguard
I’ll be presenting at Veeam on Tour
I will be hosting Veeam Webinars on Host-based Backup and Restore for Virtualized Active Directory Domain Controllers
What's New in AD DS: Active Directory Recycle Bin
Veeam Explorer for AD and AD Recycle Bin enable
Backing up Domain Controller: Best practices for AD protection (Part 1)
How to recover a Domain Controller: Best practices for AD protection (Part 2)
Active Directory Recycle Bin in Windows Server 2008 R2
Revive Deleted AD Objects with Active Directory Recycle Bin
The Active Directory Recycle Bin
Configuring Active Directory Recycle Bin
The Active Directory Recycle Bin in Windows Server 2008 R2
Enabling the Active Directory Recycle Bin Feature on Windows 2008 R2