By Ruby Nahal, Senior Engineer
Domain Controllers and Recovery
The USN rollback issue may not be an everyday problem for some organizations, but as an Engineer at a Managed Service Provider (MSP), I have seen my fair share of the condition. The problem is global – and is caused from restores of domain controllers in multi-DC environments since the introduction of Active Directory (AD).
What does it mean and when does it happen?
Each domain controller maintains a table called a ReplUpToDateVector per AD partition or naming context. This table records the DSA GUID; High Watermark (usnHighPropUpdate) and the timestamp of the last successful replication from that replication partner for the AD partition. Whenever a change takes place in Active Directory, attributes will have the originating and local USN number incremented. This just means that this is the domain controllers keep track of to track the changes made – and where they are made – according to the time stamp.
Then a replication partner will compare its own version of the table and requests the changes that are higher than the High Watermark from the source. If the USN for the DC on the replication partners is higher than the one the DC has for itself, you are dealing with a USN Rollback.
Basically, the originating DC will disable any replication as a protection mechanism in order to avoid further damage and will go in sort of a “read only” state. This can be detected by looking for event ID 2095. The originating DC may eventually catch up with the USN and start replicating, however, any changes that happen in between will be lost.
How can we avoid this?
To avoid such conditions from happening, an AD aware backup is necessary. The following do not fall into this category because these methods do not reset the DSA invocation ID when an AD restore is executed: Cloning a VM, restoring from VM snapshot, VHD copies, VMDK copies, etc.
What to do if it has already happened?
This is not a happy condition for the AD. Problems with authentication GPO processing will start showing up once you are in USN rollback condition. The easiest way out of this predicament is to restore from a system state backup because that is AD aware.
The not-so-easy method is to reinstall the Active Directory on the domain controller, which basically means transfer FSMO rules, demote DC, uninstall AD, reinstall AD, and promote DC. Not so much fun -especially when any of the above steps have to be forced because that introduces another step in all of this – metadata cleanup.
Changes with Server 2012
Windows Server 2012 virtualized Domain Controllers (Virtualized with hypervisors that support VM-Generation ID) can now be restored from snapshots without causing issues like USN Rollback. Additionally, virtualized domain controller snapshots restore resets the DC’s unique Invocation ID, discards the local RID pool and non-authoritatively restores the SYSVOL folder.
This means that accidentally restoring a snapshot is no longer an unsafe operation on domain controllers. This is quite an improvement from previous versions of Windows servers and will take away a lot of pain caused by restores from faster recovery methods like snapshots.