Running a Disaster Recovery Test That Proves Something: Tabletop, Partial and Full Failover

A plan nobody has run is a guess

Most organizations have a disaster recovery document. Far fewer know whether it works. The backup job reports success every night, but nobody has restored a whole system from it, nobody has timed the steps, and the person who wrote the runbook left two years ago. The first real test then happens on the worst day of the year.

A disaster recovery test is a planned way to find those gaps while nothing is on fire. The point is not to pass. The point is to learn what is wrong while it is cheap to fix, and to show the business, with evidence, what recovery really takes.

Three sizes of test

The U.S. National Institute of Standards and Technology describes the main exercise types in its contingency planning guide, SP 800-34 Rev. 1, drawing on SP 800-84. In plain words:

• Tabletop. People meet, a facilitator describes a scenario, and the group talks through who does what. It is discussion only; no equipment is touched. It is the cheapest test and the best one for finding unclear roles, missing contacts and decisions nobody owns.
• Functional or partial. People do their actual jobs in a simulated environment: restore a database to an isolated network, rebuild a server from backup, switch one application to its standby copy. It proves that the procedure and the tools work, one piece at a time.
• Full-scale failover. The production workload really runs from the alternate location, then returns. It is the only test that proves the whole chain, including network, identity, DNS, licenses, people and the return trip. It is also the one with real risk, so it needs the most preparation.

NIST's guide links the size to the importance of the system. For low-impact systems it says a tabletop at an organization-defined frequency is sufficient. For moderate-impact systems it calls for a functional exercise that includes recovery from backup media. For high-impact systems it calls for a full-scale exercise that includes failover to the alternate location and a full recovery back to a known state. You decide the frequency; the guide ties the depth to the consequence.

Match the test to the system

A sensible program uses all three sizes. Run a tabletop for every important scenario each year: a ransomware event, loss of the main site, loss of a cloud region, loss of a key supplier. Run partial restores on a rolling schedule so that every critical system is restored from backup at least once in the cycle. Reserve full failover for the handful of systems where the business has set a short recovery time and where a failure would hurt most.

Start with the tier list from your impact analysis. The systems with the tightest recovery time objectives earn the deepest tests; the rest can share a tabletop. If you have not set those objectives yet, do that first, because a test with no target cannot fail and so cannot teach.

Write the pass mark before the test

The most common weakness in a drill is that success is decided afterward. The team recovers most of what it wanted and calls it a pass. Avoid that by writing criteria in advance and getting the business owner of each system to sign them.

• Scope: which systems, which data, which users, which site.
• Time: the recovery time objective for each system, and the moment the clock starts. Starting the clock when someone notices, not when the failure began, hides the detection delay.
• Data: the recovery point objective, shown by checking the newest record in the restored data against the time of the simulated failure.
• Function: not 'the server is up' but 'a named user signed in, ran a named transaction and saw correct data'. Include identity, DNS, certificates, licenses and printers or phones if the business needs them.
• Security: the restored system has current patches, the same access controls and logging, and the restore did not reintroduce the problem that caused the disaster.
• People: the plan could be followed by the person on call, using only the document, without calling the person who wrote it.

Make the scenario honest

A scenario that says 'the primary server fails' is too polite. Real disasters take out the thing you were counting on: the password manager sits in the failed site, the runbook is on the file server that is encrypted, the one engineer who knows the firewall is on vacation, the backup console needs the same identity system that is down. Add one such complication to each exercise. A facilitator can introduce it mid-way, which also shows whether the team improvises well.

For ransomware scenarios, assume the backup repository may be reachable by the attacker and ask what is truly isolated or immutable. For site-loss scenarios, assume the building and the people near it are unavailable. For a tabletop that includes executives and the board, plan on a separate session with a decision focus; a different article covers that format.

What to record

Evidence is what separates a test from a conversation. Appoint one person who is not doing the recovery to keep the record. At a minimum capture:

• The scenario, the date, the participants and their roles, and the version of the plan used.
• A timeline with real times: disruption declared, decision to fail over, first system usable, last system usable, business validation finished, return to normal.
• For each system, the achieved recovery time and the data age against the objective, with pass or fail.
• Every deviation from the written procedure, every workaround and every step that was unclear.
• Surprises: missing credentials, expired licenses, firewall rules that were not in the document, network ranges that clashed.
• Screenshots or log extracts that show the restore result, so the record does not rest on memory.

Close the loop

Within a week, hold a short review. Turn each finding into an action with an owner and a due date, and put those actions where the team tracks other work. Update the plan itself, and mark the date and the version. NIST's guide says all tests and exercises should include a determination of the effect on operations and a way to update and improve the plan as a result; the update is part of the test, not an optional extra.

Share a one-page result with the business owners: what was tested, what the objectives were, what was achieved, what will change. If the achieved times missed the objectives, that is a business decision, not a technical embarrassment. Either spend to close the gap or agree a longer objective in writing.

Keep failover safe

A full failover test can cause the outage it simulates. Reduce the risk: pick a quiet window, agree a go or no-go point, keep a tested way back, tell customers or staff if they could notice, and make sure that production data written during the test is not lost when you return. Many teams run the first full test against a copy of the environment in an isolated network, then move to a live failover once the procedure is stable.

A simple yearly rhythm

• Quarter one: tabletop on the most likely scenario with the managers who would run it.
• Quarter two: partial restores of the top-tier systems into an isolated network, timed.
• Quarter three: a full failover or a large functional exercise for the systems with the shortest objectives.
• Quarter four: tabletop with executives and the people who talk to customers, then update the plan and the objectives for the year ahead.
• Any time: after a major change, test the changed part again.

The aim is a calm, boring result: the team has done this before, the times are known, and the surprises were found in a drill.

Sources

• NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems (NIST CSRC)
• NIST SP 800-34 Rev. 1, full text (NIST)
• NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities (NIST CSRC)
• Exercises (Ready.gov)

This article is general guidance, not legal advice. Adapt it to your organization, and ask your counsel about employment, privacy and contract questions before you rely on it. Last reviewed November 9, 2026.

Next article

Next · RegulationK-12 Student Data Security: FERPA, COPPA and CIPA in Plain Words, and How to Vet an Edtech VendorA school district runs dozens of apps that hold student records, and each one is a door. Three federal rules shape the conversation: FERPA on education records, COPPA on services for children under 13, and CIPA on filtering for schools that receive E-Rate discounts. This is a plain-language map of what each one covers, how they differ, and a vendor checklist built from the regulations and Department of Education guidance.8 min read

Related reading

• InfrastructureSetting RTO and RPO With the Business: Impact Analysis, Tiers and Cost Trade-Offs8 min
• InfrastructureHow to Run a Microsoft 365 Restore Test: What to Measure, What Retention Does Not Cover8 min