In this post I cover Disaster Recovery tests, what they are, what’s their purpose, and why some organizations and institutions end up never doing them and being caught off guard whenever something bad happens.
A bit of context
After the ANCPI hack that took down Romania’s land registry, Andrei Avădănei, CEO of Bit Sentinel and founder of DefCamp, published on LinkedIn a detailed proposal for a national offensive security program.
It covered pentesting frameworks, vulnerability disclosure, continuous monitoring, and accountability measures. The proposal was thorough, logical, and exclusively focused on prevention and detection.
I left a comment suggesting one addition: mandatory disaster recovery simulations.
Can institution X recover after their entire production environment is encrypted? If so, how long does it take and what data is lost? Are there backups? And if yes, are they actually viable, or are they Schrödinger’s backups, where you only find out whether they work at the exact moment you need them?
This exchange made me realize that organizations, especially in the public sector, rarely consider doing disaster recovery tests.
What is a Disaster Recovery test?
A DR test is, as the name implies, a test of an org’s ability to have their infrastructure back in a functional/production-ready state after a disaster.
It’s basically a rehearsal of all the steps required for your org to get back online.
Disaster recovery tests also offer a better way to gauge, adjust, and work towards more appropriate RTOs* and RPOs*.
*RTO = Recovery Time Objective, which is the time needed to bring one or more systems back to a functional state.
*RPO = Recovery Point Objective, this is the point in time to which one or more systems can be recovered.
Or, in simpler terms, how much data can be lost in case of restore/disaster recovery.
Some folks might conflate having backups alone with having a recovery plan, but, while having backups is a vital part of any recovery plan, a recovery plan consists of way more than just backups.
A disaster recovery test generally involves:
- Documenting your environment, assigning a priority to every system and component involved.
For example: manufacturing-related systems have a higher priority than HCM systems because the company’s production capabilities rely on those manufacturing systems. - Identifying dependencies and service tiers.
- Simulating a failure scenario (data center outage, hardware loss, ransomware encryption, storage corruption).
- Depending on the type of infrastructure, the recovery can consist of: failing over to another data center or to secondaries in the same location, restoring on secondary hardware or restoring in place.
- Measuring how long each step took against predefined targets agreed upon with stakeholders and business.
- Adjusting either the recovery targets and/or the infrastructure and/or the DR plan to make the recovery goal achievable.
Why some orgs tend to overlook DR tests
From what I’ve seen, organizations tend to avoid doing DR tests unless they are legally or contractually obligated to do so.
The reasons might vary from case to case.
Complexity aversion
Sometimes it’s an aversion towards complexity and the work involved.
DR tests tend to touch every layer:
- network
- storage
- compute
- applications
- DNS
- AD
- firewalls
- databases
Coordinating all that is a non-trivial task, so it can get deferred for a later time that might never come.
And when it is attempted, scope creep can turn a focused exercise into a multi-week project that nobody wants to repeat.
Business disruption
IT and the business might not always see eye to eye on why DR tests are necessary.
And this can lead to a conclusion like “We can’t take production offline for your silly test.”.
As a result, the disaster recovery test never happens, and, when an actual disaster strikes, the first attempt at recovery is also the first time something like that ever happens.
In some cases this can be due to a failure of communicating to the business in a way that clearly outlines why it’s preferable to have a brief, controlled, production outage in order to be better prepared for when an actual outage comes.
In short: you either disrupt the business on a schedule you control, or an attacker/failure will disrupt it for you when you least expect it.
Compliance-driven apathy
If the audit checklist says something like “DR plan exists”, then that’s what gets checked.
But chances are, nobody will feel obligated to check:
- If the plan has ever been executed.
- Whether the runbooks are current.
- If the people named in the plan still work there.
- If any new systems have popped up in the meantime or if any old ones have been retired.
The box just gets ticked and everyone moves on.
Magical thinking aka “that won’t happen to us”
It kind of boggles my mind that, after all the breaches we’ve seen in the past 2 decades, there are still people out there who treat this as a matter of “if” instead of “when”.
Hardware fails, systems are vulnerable, even large cloud providers have outages.
If something bad can happen, chances are it will happen eventually. Might as well be prepared for it.
Hardening, patching, EDR, SIEM, zero-trust, zone-redundant and geo-redundant infrastructure: all of these create a psychological comfort zone. And that comfort zone can lull organizations into believing recovery planning is unnecessary.
It’s the cybersecurity equivalent of wearing a seatbelt but never checking if the airbag works.
Failure as taboo
In some cases, people might be tempted to treat DR tests and planning as planning for failure and a sign of lack of confidence in the team.
When you’re doing a fire drill it’s not like you expect a fire to start, you just want to be prepared in case that ever happens. The same goes for DR tests.
It’s also the fact that DR testing lacks the glamour of other cybersecurity work.
Nobody gets promoted for recovering from a theoretical disaster. But doing pentests, deploying a shiny new EDR tool, standing up a SIEM, or presenting a zone/geo-redundant architecture roadmap, those are visible, measurable, budget-justifying wins.
DR testing is the unglamorous insurance policy that nobody notices until there’s an actual disaster.
The benefits of disaster recovery testing
RTO and RPO stop being theoretical
Disaster recovery tests ensure that your RTO and RPO are actually based in objective reality and not some idealized made-up numbers.
Without actually testing, your RTO is a number someone wrote in a document that may or may not be realistic.
After a test, it becomes a measured value.
You discover that your “4-hour RTO” is actually 11 hours because the database restore alone takes 6 hours and nobody accounted for the app server needing to be rebuilt from scratch because it was never backed up.
Shedding some light on shadow IT
DR tests tend to reveal dependencies that don’t exist in any documentation.
When you ask “What needs to be online for payroll to run?” and someone says “Well, Janet pulls data from the SQL Server instance used by our time tracking software into an Access database on her desktop where she calculates the rates and salaries.” you’ve just discovered a critical path that no CMDB knows about.
Muscle memory
The first time someone restores a database under pressure, they’re reading documentation. The third time, they’re doing it from memory and catching errors before they cascade.
DR testing builds incident response reflexes that pay off during actual outages.
The people who’ve done it five times stay calm. Folks who’ve never done it might get caught up in the chaos.
New automation opportunities
Another bonus of DR tests is that they tend to expose manual processes that nobody questioned because “we’ve always done it this way” and are prime targets for automation.
When you’re walking through a recovery scenario and someone says “well, first I log into the server, then I open SSMS, then I right-click and restore, then I manually reconfigure the logins, then I…” you’ve just found an automation candidate.
Anything that’s done by hand during recovery is a bottleneck under pressure.
A DR test turns those slow, GUI-heavy and click-intensive procedures into a priority list of things you should automate.
Restore sequences, permission syncing, database mail configuration, agent job recreation, service accounts, application configurations, etc. The kind of tedious multi-step work that eats precious minutes during an incident and is prone to human error when everyone is racing against the clock to get prod back online.
The neat part here is that the benefits of these automation efforts are two-fold:
- You speed up real recovery tasks which help bring everything back online faster when a real disaster strikes.
- It makes your subsequent DR tests faster and less painful which means you’re more likely to do DR tests again.
System prioritization
Disaster recovery testing forces conversations like “if we can only bring up 5 systems in the first 5 hours, which 5 would those be?”.
Having that conversation before a real disaster is productive and helps build a recovery plan that’s actually useful. Having that conversation during an actual disaster is more chaotic than it needs to be. Especially since every app owner/stakeholder tends to think that their app/system is the highest priority.
As a result, DR testing helps create a ranked recovery order based on business impact, not on whoever screams the loudest during a crisis.
As an example, if your company’s main activity is manufacturing related: In a real DR scenario, the systems used for manufacturing, order taking, billing and payment processing, should be the top priorities. Not some file share server that’s used to archive old quarterly sales reports or a dashboard that’s only checked once every 2 days.
Conclusion
In today’s cybersecurity landscape, periodic DR tests should be the norm.
Especially for public institutions and organizations managing critical infrastructure.
While the ANCPI hack was a wake-up call, the next attack will still find most institutions underprepared.
Prevention and detection are vital, but they can’t help you once someone gets inside your environment and starts encrypting everything.
Without actual disaster recovery tests, your DR plan is nothing more than fanfiction about your infrastructure and systems.