India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
DR Testing: Timed Drills That Prove RTO and RPO with Evidence
DR testing is the planned exercise of recovery capability, from tabletop walkthroughs to live failover, to measure whether systems actually recover within their RTO and RPO and to find what breaks. NTSPL designs annual test calendars, runs timed drills on AWS, Azure and Google Cloud, and produces evidence packs and fix lists after each test.
Delivered on
In scope
Drills Designed to Find Failures, Not to Pass
Most DR tests are designed to pass. The team picks a quiet weekend, pre-stages resources, runs only the components that are known to work, and reports success without timing each step or checking data loss. Auditors receive a screenshot and a sign-off. Then a real event reveals what the drill avoided: a dependency in another team's account, a partner integration that rejects the DR IP, a decision that took an hour, or a replica that was 40 minutes behind. In India, regulators increasingly ask not whether DR was tested, but what the measured RTO was and what was fixed afterwards.
NTSPL runs DR tests as measurements. Every drill has a hypothesis, success criteria set in advance, a timekeeper logging each step, a data-loss check comparing last committed transactions, and an observer recording decisions and confusion. Tests escalate across the year from tabletop to component to isolated failover to live traffic switch, and chaos experiments inject failures responders do not know about in advance. Each drill ends with a scored report, a fix list with owners, and evidence formatted for auditors.
When Your DR Tests Are Not Telling You the Truth
Signals you need DR Testing
- Your last DR test report says 'successful' but contains no step-by-step timings or measured data loss.
- DR has not been tested with live traffic in the last 12 months, or ever for some Tier 1 systems.
- A regulator, auditor, customer or cyber insurer has asked for DR test evidence with RTO and RPO achieved.
- Systems, teams or vendors have changed significantly since the last test, such as a migration, new CBS version or reorganisation.
- You want to start controlled chaos experiments but need guardrails and a progression plan.
When to choose something else
- There is no DR capability yet to test. Build a pattern first; a test of nothing is a design exercise.
- Runbooks do not exist. A drill without runbooks mostly tests improvisation; write runbooks first through DR Runbooks, then test them.
- You only want daily backup restore checks. Routine restore testing is part of backup operations rather than a DR test programme.
The Five Drill Types in an NTSPL Test Programme
Tabletop exercise
A facilitated scenario walkthrough with technical and business participants, testing decisions, roles and communications without touching systems.
Component test
Recovery of a single layer such as database promotion, DNS switch or backup restore, timed and verified in isolation.
Isolated failover
Full stack recovery into an isolated network in the DR region while production continues, verifying the recovered system end to end.
Live failover
Production traffic moved to DR for a defined period, including partner integrations and real users, then failed back.
Chaos and surprise drills
Unannounced or semi-announced fault injection, such as instance termination, zone isolation or dependency failure, with guardrails and stop conditions.
How DR Testing Works, Step by Step
DR Test Cycle: Hypothesis, Execution, Measurement, Fix
Each test starts with a written hypothesis and success criteria, runs with independent timekeeping and data-loss measurement, and ends with a scored report and fix list. Fixes feed the next test, and evidence is stored for audit.
Select any component
DR Test Cycle: Hypothesis, Execution, Measurement, Fix: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Fault Injection and Test Tooling per Hyperscaler
Fault Injection Service runs experiments with CloudWatch alarm stop conditions, and Elastic Disaster Recovery launches non-disruptive drill instances while replication continues.
Site Recovery test failover recovers into an isolated network without affecting replication; Chaos Studio injects faults such as VM shutdown or network disruption against selected targets.
Google Cloud has no single fault injection service, so NTSPL scripts controlled faults such as instance group shrinks, replica promotions and firewall isolation with Monitoring alerts as stop conditions.
Choosing Drill Scope, Realism and Frequency
Test frequency by tier
Tier 0 and Tier 1: quarterly component tests, at least one isolated failover, and one live failover a year, with many regulated entities choosing two. Tier 2: one isolated failover a year and component tests twice a year. Tier 3: an annual restore test. Tabletop exercises for leadership at least twice a year, since people and roles change faster than systems.
Isolated versus live failover
Isolated failover is safe and frequent but misses partner integrations, real user load and DNS behaviour. Live failover catches those but carries customer risk. Run isolated tests to fix technical issues first, then a live failover during a low-traffic window with business notification, a rollback plan and stop conditions. Do not start with a live failover on a system that has never passed an isolated one.
Who runs the drill
The people who will respond to a real event, usually the on-call rota, should execute the drill. Designers and vendors observe. Rotate the incident commander role so that more than one person has done it. For at least one drill a year, run with a deputy as commander and key people deliberately unavailable.
Introducing chaos experiments
Start with experiments in non-production, then production faults with small blast radius such as a single instance or zone during business hours with the team aware. Progress to semi-announced game days where responders know the window but not the fault. Always define stop conditions using alarms on customer-facing metrics, and halt immediately if they trip.
Setting Up a 12-Month DR Test Programme
Test baseline and calendar
Previous test records are reviewed, a scope per tier is defined and a 12-month calendar avoiding business peaks is approved.
Hypotheses and measurement design
Success criteria, timekeeping method and data-loss measurement are defined for each scheduled drill.
First tabletop and component tests
Leadership tabletop and component tests are run to surface decision and runbook gaps cheaply.
Isolated failover drills
Tier 1 services are recovered into isolated networks in the DR region, timed and scored.
Live failover and fix cycle
A live failover is executed for the most critical service, findings are fixed, and the calendar continues with retests.
What you receive
Measurements Captured in Every DR Drill
| KPI | Typical target |
|---|---|
| Tier 1 services with measured RTO in the last 12 months | 100% |
| Achieved RTO versus target | Within target, or a dated fix item |
| Achieved RPO measured by transaction comparison | Within target for every drilled service |
| Drill findings closed before next drill | 90%+ |
| Share of drill time spent on decisions | Reduced drill over drill; under 20% of RTO |
Drill Risks and How NTSPL Keeps Tests Safe
| Risk | How NTSPL prevents it |
|---|---|
| Live drill causes a real outage | Live failovers follow a passed isolated failover, run in low-traffic windows with stop conditions, business notification and a tested rollback. |
| Drills staged to pass | Success criteria are set in advance, an independent timekeeper logs every step, and pre-staging is recorded and disallowed beyond what normal operations provide. |
| Isolated test environment leaks into production | Isolated networks have no routes to production, test instances are tagged and auto-deleted, and outbound integrations are disabled or pointed at sandboxes. |
| Findings never get fixed | Each finding has an owner and retest date, and open findings are reported to the risk committee alongside the drill result. |
Tabletop exercise vs live failover drill: which one do we need?
Both, for different failures. A tabletop tests people: whether leaders know who declares, how the business operates during recovery, how customers and regulators are informed, and how decisions are made under uncertainty. It is cheap and should run twice a year. A live failover tests technology and integrations: whether systems actually start, data is current, partners accept traffic and users can transact. It is expensive and risky, so it runs once or twice a year after isolated tests pass. Organisations that only do tabletops overestimate technical readiness; those that only do failovers often stall on decisions.
Key facts: DR Testing
DR Testing in Practice
2 DR Testing scenarios, from Event and cinema ticketing and Municipal water supply and sewerage utility, plus related platform case studies.
Ticketing platform's first honest drill runs 2.3x over RTO, the third meets it
City water utility runs its first tabletop and finds that billing DR was not the gap
Telecom VAS provider cuts daily pages from 2,300 alerts to 9 actionable incidents
IT services firm brings 43 Azure subscriptions under one policy hierarchy
DR Testing: Frequently Asked Questions
How much does a DR testing programme cost?
Setting up the programme and running the first cycle of drills takes about 12 weeks and 30-50 consultant days for 5-10 Tier 1 services. Each later drill costs a few consultant days plus cloud run cost of the DR environment during the test, often INR 10,000-80,000 per drill depending on scale.
Do RBI and SEBI require DR drills?
Yes. RBI expects regulated entities to test BCP and DR periodically and report outcomes to the board or its committees. SEBI expects market participants to conduct DR drills, and market infrastructure institutions have specific drill and live-from-DR requirements. NTSPL formats evidence for your sector's inspection expectations.
Can we test DR without customer impact?
Mostly, through isolated failover and component tests, which run while production continues. Live failover does involve customers, but in a low-traffic window with tested rollback, impact is usually a few minutes of reconnects. Many teams also use the live drill to validate planned maintenance procedures.
What is a good result for a first drill?
Finding problems. First drills commonly exceed RTO by 50-200% because of missing dependencies, slow decisions or stale runbooks. The value is the fix list. By the second or third drill, most teams reach their targets, and the focus shifts to keeping them as systems change.
Should we include ransomware recovery in DR tests?
Yes, as its own drill. Test restoring from locked or isolated backups into a clean room, choosing a recovery point before a simulated intrusion, and getting security sign-off. It typically takes much longer than failover, and leadership should see the realistic number.
How do we test DR for SaaS and third-party services?
You cannot fail them over, but you can test your degraded mode: queuing transactions, switching to a backup provider or manual workarounds. Ask providers for their own DR test reports and include their contractual RTO in your drill assumptions. Their gaps go on the findings list.