DTAPR – Is it worthwhile to add a Ransomware Recovery environment?

Reading Time: 4 minutes

Development | Test | Acceptance | Production | Recovery

Serious IT environments don't just have a test environment. They also have development, acceptance and/or production environments. 🤡

For applications, having a development, test, acceptance (on-premises) and/or staging (typically cloud) implementation or instance seems common. For infrastructure, however, it is not. The availability, confidentiality and integrity of many Active Directory environments needlessly suffer because of this choice, but that is not the topic I want to discuss today. Today, in the context of World Backup Day 2024, I want to talk about the idea of dedicated Recovery environments.


The number 1 digital business risk

This year, but otherwise next year, business will see ransomware and other abuse of business IT resources as the biggest risk of doing business digitally. Yet, we don't seem to adequately assess this risk. It's understandable. Most of use have not experienced a catastrophic ransomware incident… yet.


Typical DTAP processes

In terms of processes, (logically) separated development, test and acceptance environments resemble the production environment in terms of components necessary to develop, test or accept changes, within a certain scope. Often, this scope is limited to one or more business applications. To further reduce the risk of rogue changes, development, test and acceptance (DTA) environments can also be provisioned with infrastructure as scope.

When looking at Active Directory, changes to objects, attributes, settings and Domain Controllers can be developed (to develop the code to use), tested (to ascertain the proper technical outcome) and accepted (to ascertain the proper business outcome) before applied to the production environment. For all intents and purposes, when an organization doesn't have any non-production environment, they basically only have a test environment.

Non-production environment can be provisioned from code (typically useful for development environments) or from backups (typically useful for test and acceptance environments, but also to test backups). These environments can be spun up and decommissioned at any time to suit technical and/or business needs. Veeam's SureBackup feature greatly simplifies spinning up Active Directory environments from backups.


The need for a Recovery environment

The vast majority of successful ransomware and other large abuse cases of business IT, features malicious use of Active Directory to elevate privileges, evade detection, persist and ultimately to propagate. Active Directory is the tool of choice for attackers. Active Directory is also the critical resource that organizations require to eventually return to business as usual after a catastrophic availability incident.

At these times, however:

The current IT resources cannot be used

Forensic and other partners require access to these resources to determine the initial attack vector and remediate any entrance points before returning to business. Otherwise, the IT resources would just be ransomed again.

The current IT resources cannot be trusted

Potentially dangerous Domain Controller should not be deployed on potentially dangerous virtualization platforms, on hosts with potentially dangerous firmware versions on potentially dangerous networks with potentially dangerous hosts connected to ‘em. Restoring from a potentially dangerous backup server can also not reliably used. All the pieces of the chain need to be checked.

When the initial entry point is not cut off, it will be reused

Simply restoring backups to hosts to replace encrypted hosts is not going to cut it. The initial entry point, and all the other means abused by attackers on their attack path, need to be remedied. If the attack path is not remedied, the environment will be re-encrypted and the organization would eventually realize that forensic research needs to take place to –eventually– avoid re-encryption.

Backups may also be encrypted

In typical backup scenarios, backups to immutable storage are copy backups. This means that the backup that is sent to immutable storage is a copy of the backup that is stored with the backup server and thus potentially in scope of attackers. In this attack scenario, the immutable period merely dictates the time-out that attackers would need to take (also known as dwell time) before starting the next phase of their nefarious plan.

15 days retention is likely not going to be sufficient to detect adversaries, 180 days immutability may prove to be sufficient to provide a point in time (RTO) to go back to safely, but storage costd may skyrocket beyond the maximum storage offered by your pod-based solution.

The two faces of immutable backups

Immutability for backups is great… to make sure organizational information is always available to restore from. This negates the first (and foremost) play of ransomware actors: blackmailing organizations to get access to their now encrypted data. However, ransomware actors increasingly adopt a triple-play strategy (or triple-pay, if you're more cynic about this topic) where the ransomware actor would not only blackmail the organization to gain access to their data, but also ransomware the organization to have others not gain access to the ransomware actor's copy of the organizations data that they siphoned off, before encrypting the data. The same data can then be reused by the ransomware actor (or a different actor in the same ransomware ecosystem) to blackmail individual customers of the organization to prevent unauthorized access to the data processed by the organization on their behalf.

Immutable backups are always-there backups. With the right information (the backup master key and a storage access key) an immutable backup is the ideal source of organizational data for a ransomware actor. With immutable backups typically stored in cloud storage and cloud access and/or activity logs not centrally stored, processed, detected or investigated, a ransomware actor can siphon off data from the cloud storage provider. Most often, this would mean a higher throughput for them, too.

Access to immutable backups needs to be restricted, governed and monitored.


Recovery environment to the rescue!

So, what do you do when you find out your networking environment has been encrypted and cannot be used for anything until your forensic department and/or partners give the green light?

You use a dedicated recovery environment that is disconnected from the Internet and the backup server in that environment to restore all Domain Controllers, so that the same forensic department and/or your forensic partner can give a green light on them. In the pitch-black worst case scenario, this typically shaves off one to two weeks towards a safe restore as the forensic actions towards the encrypted platform don't stand in the way of forensic actions towards Active Directory. They can now be performed in parallel.


Is it worthwhile?

To answer the question if a dedicated ransomware recovery environment is wothwhile, an organization needs to weigh the benefits of being able to restore no matter the circumstances the organization faces against the costs of maintaining such an environment and extending current DTAP processes.

For organization who are already invested in DTAP processes, I think it's worthwhile.

Author: Sander Berkouwer

Sander Berkouwer is the author of the Active Directory Administration Cookbook, speaker and blogger at DirTeam.com and ServerCore.net. He is awarded Microsoft MVP, Veeam Vanguard and VMware vExpert. Since 2009, Microsoft has awarded Sander with the Most Valuable Professional (MVP) award. Since 2016, Veeam has awarded Sander with the Veeam Vanguard award.