When installing updates, there is always the risk of rogue updates; updates that break functionality, unannounced, unexpected and unsettling. Microsoft is currently researching such a possible side-effect with the January 11, 2022 updates on Active Directory Domain Controllers.
About the issue
Domain Controllers may reboot unexpectedly and keep rebooting. Event ID 1000 is triggered right before these reboots citing that lsass.exe had failed with stop error 0xc0000005 (access violation), status code -1073741819 and pointing to msv1_0.dll as the culprit.
The Local Security Authority Subsystem Service (LSASS) is responsible for enforcing the security policy on the system. It verifies users signing in to a Windows or Windows Server, handles password changes, and creates access tokens. It also writes to the Windows Security Log. Forcible termination of lsass.exe will result in a restart of the Domain Controller. The restarts are the actual recovery process, not the problem.
Unconfirmed details and symptoms
At this time, there are a couple of unconfirmed details and symptoms about this issue:
- Domain Controllers running Windows Server 2012 R2, Windows Server 2019 and Windows Server 2022 seem most affected.
- Domain Controllers in environments with Exchange Servers seem most affected.
- Read-only Domain Controllers seem unaffected.
About the updates
The following updates are available for Windows Server installations as part of the January 11, 2022 updates:
- Windows Server 2012: KB5009586
- Windows Server 2012 R2: KB5009624
- Windows Server 2016: KB5009546
- Windows Server 2019: KB5009557
- Windows Server, version 20H2: KB5009543
- Windows Server 2022: KB5009555
Active Directory admins experiencing continually rebooting Domain Controllers share that they have stopped the reboots by disconnecting the network connection and uninstalling the January 11th, 2022 update from these systems. They rebooted the systems and after this reconnected the network connection.
When installing security updates only on Domain Controllers running Windows Server 2012 R2, uninstalling KB5009595 also seems sufficient.
To uninstall these updates, run the following command line:
Windows Server 2012 R2: wusa.exe /uninstall /kb:5009624
Windows Server 2019: wusa.exe /uninstall /kb:5009557
Windows Server 2022: wusa.exe /uninstall /kb:5009555
I’m not a fan of not having critical updates installed, but in this case I feel it may be wise to wait 10 days before installing the January 11th, 2022 updates on Domain Controllers. My experience is that serious problems like the above problem are addressed within that timeframe.
Microsoft pulls new Windows Server updates due to critical bugs
Windows Server: January 2022 security updates are causing DC boot loop
January updates causing unexpected reboots on domain controllers : sysadmin
I wish I could say this worked on our Windows Server 2012 R2 Domain Controller….. after uninstalling the update the Domain Controller still reboots with the same errors.
I had this issue on Windows Server 2012 R2,
Windows Server 2016 Standard and Windows Server 2016 Datacenter domain controllers.
Following uninstall instructions helps.
Thanks a lot!!!
Lifesaver! Thank you
This affected Azure Active Directory Directory Services instances a few days ago, leaving our Azure VMs without a secure channel for a day. Needless to say someone c*cked up and Microsoft is playing mum
also remove update KB5009595 (2012r2)
I installed the patch on one of our domain controllers, on Friday. Took a snapshot prior to installing the patch. No reboots on Friday, Saturday and Sunday. I removed the snapshot on Monday, as the DC did not reboot once. Took a snapshot of a second DC, and patched the DC. Performed the same procedure on a third DC. 5 minutes after the third DC was patched, all DC's started to reboot unexpectedly. After 2 executive reboots, we reverted the snapshot's where available, uninstalled the patch on the DC which was patched first, which no longer had a snapshot. The uninstallation of the patch took almost 30 minutes to complete after the server was rebooted.
Had this issue on 2 DC's over the weekend. Only figured it out this morning by uninstalling KB5009624 on our Windows 2012 R2 servers. It's been up and running for 3 hours now without reboots.
Talk about causing a fire drill!
Thanks Sander, great information as always.
Only wish I had read it on the 14th!
Pingback Microsoft has released out-of-band updates to address Domain Controller boot loops - The DirTeam.com / ActiveDir.org Weblogs
Had this affect multiple school sites Domain Controller's over the last few days. Luckily with all but one, uninstalling the problem update fixed the issue.
With the one that this didn't work (Suspect the issue in this article started a cascading into several other issues), the only solution was a complete restore from backup image.
My 2016 Forest seems to not be affected- no 1074 lsass.exe crash system events. However, my 2019 Forest did. uninstalling the kb5009557 took forever (30 minutes like mentioned above) to get past the 100% updates please wait on reboot but during that time, all authentications worked. I was finally able to check the update history and it is no longer on them. Servers are faster, too since the lsass.exe is not taking full CPU.
Thank you for the post, I removed KB5009546 on Domain Controllers running 1Windows server 2016.
Microsoft have released the following patch to address the issue: KB5010794.
It shows up as optional patch.
KB5010794 does not appear to be available yet for WSUS.
FYI – Reboot in Safe Mode, with networking – that will stop the reboot.
Uninstall the update, then reboot.
Once back online, Follow these steps to stop the automatic updates: https://rdr-it.com/en/howto/windows-server-2016-2019-disable-automatic-updates/
This worked for us.
Thanks a lot, it works!!!
Ufff we didn't had email (exchange server and other services that need authenticate with AD).
I have 2 Domain controllers (one of them vitual). The main DC receive (physic) the update the last january 15th and we hadn't detect the problem until today Wed Jan 26th because I restart the Virtual DC and they started to retart both unexpetedly when still the virtual hadn't the winupdate. I think the main DC sent a signal to restart the virtual.
FIX: I restarted them without the cable network connected to work in them and I applied the uninstall in the physic server and worked normally again both servers.
Getting the same problem on Windows 2016. Uninstalling KB5009546 now. The symptoms we get are a daily reboot at 12:21am every day and auto restarts once. Update KB5010790 is repeatedly trying to install and failing. not sure if its related but may help someone. Thanks
Thank you so much. I had this issue on our two Win 2019 DC's (also with exchange), on doing a routine reboot. Uninstalling KB5009557 fixed it.
I will never again do updates or reboots on a Sunday evening…
Good Job !!