A distributor tests its disaster recovery for the first time: documented RTO and RPO, a failover to a second region and what broke along the way
A wholesale distributor of about 350 staff with a main office and two warehouses in the southeastern United States, a leader in its segment.
- Scale
- ~350 staff, 14 servers replicated to a second region, 1 ERP database, RTO 4 hours, RPO 15 minutes, 7 defects found and fixed
- Engagement
- 4 months: 6 weeks of documentation, a rehearsal, then a one-day failover test
The challenge
The distributor ran its ERP, file shares and mail relay in a single data center and replicated them to a second region, but it had never tried to run from the copy. The plan was a document written three years earlier. Leadership assumed a recovery would take an afternoon, a hurricane season was starting, and the insurer had asked when the plan was last tested.
What our team delivered
- A business impact review with the department heads that sorted systems into tiers and wrote the targets down: the ERP, order entry, warehouse scanning and identity as Tier 1 with a recovery time objective (RTO) of 4 hours and a recovery point objective (RPO) of 15 minutes, file shares and reporting as Tier 2 within 24 hours
- A recovery runbook with a named owner for each step, the dependency order (identity and DNS first, then the database, then the application servers, then the warehouse scanners), the credentials location, the call tree and the criteria for declaring a disaster and for failing back
- A rehearsal in an isolated network in the second region, followed by a one-day failover test on a Saturday: production writes frozen, a final replication check, the second-region copies started, DNS records with a 60-second TTL switched, IPsec tunnels from both warehouses to the second region confirmed and a sample of staff running real orders
- A timed log of each step, with the data loss measured by comparing the last transaction in the ERP database against the replicated copy, so the real RTO and RPO were measured, not estimated
- Seven defects found by the test and fixed within three weeks: the second-region domain controller held a stale replica and needed an authoritative restore; the ERP license was tied to the primary server's hardware ID; the warehouse scanners pointed at a hard-coded IP address; a firewall rule from the warehouse tunnels to the application subnet was missing; a service account password had been rotated in production but not in the recovery runbook; the mail relay's SPF record did not list the second region's address; and the backup agent in the second region was not enrolled in monitoring
- A second test eight weeks later that confirmed each fix, then a calendar of one tabletop and one technical test every year, plus a retest after any major change to the ERP or network
Result
The first test took 6 hours 10 minutes to reach working order and lost 9 minutes of data, outside the 4‑hour target. After the seven fixes, the second test ran the same steps in 3 hours 25 minutes with 8 minutes of data lost, inside both targets. The distributor now answers its insurer and customers with a dated test report. The figures are illustrative and a modeled outcome, and results differ with each environment.
Get this for your company
Related services with fixed prices and a written scope.
Emergency package: security engineer in 1 hour
Is your company in a cybersecurity crisis? Purchase this package and a security engineer will call you within 1 hour.
Callback within 1 hour, 24 hours a day, 365 days a year
$1,950 per incident
Insurance-ready bundle
The three things cyber insurers and clients ask for most: an external assessment, a phishing simulation for 25 users and a disaster recovery plan.
Assessment report in 5 business days; plan draft in 15 business days
$5,400 per bundle, phishing simulation for 25 users
Managed IT and security (MSP)
One monthly price per team member: help desk, device management, patching, backup monitoring and security basics.
Onboarding starts within 10 business days
$100 per team member, per month
Show 4 more
Initial assessment of public endpoints
What an attacker can see and reach from the internet: your domains, mail, websites, remote access and cloud exposure.
Report within 5 business days of kickoff
$1,000 per assessment
Managed security monitoring
Endpoint detection and response watched by our analysts, priced per device.
Agents deployed within 7 business days
$18 per endpoint, per month
Cloud security review
A review of one Microsoft 365, Azure, AWS or Google Cloud tenant against recognized benchmarks.
Report within 7 business days of read-only access
$2,800 per tenant
Cloud management
We run your Microsoft 365, Azure, AWS or Google Cloud environment: access, cost, patching, backup and security settings. Priced per month for one tenant.
Onboarding within 15 business days
$1,900 per tenant, per month
After you purchase, we contact you within 1 hour, any time of day or night, to start onboarding.
No payment is taken online. When we call you within the hour we confirm the scope and send an invoice (ACH, card or crypto).
Backup tiers: data lost versus time to recover
Recovery point objective (RPO) is how much recent data a restore loses; recovery time objective (RTO) is how long the restore takes. Faster tiers sit close to the production system and share its fate; slower tiers survive ransomware and site loss. The scale is logarithmic. Illustrative figures.
Failover in under a second
ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.
A DNS change takes one TTL to settle
After you change a record, resolvers keep serving the old answer until their cached copy expires. Each cache was filled at a random moment within the last TTL, so the share still serving the old answer falls in a straight line from 100% to 0 over one TTL (the time axis is logarithmic, which bends the lines). Lower the TTL a full old-TTL before a planned change. Some resolvers ignore very low TTLs, so treat these as best case. Illustrative model.
Super Intelligence data flow
Every request is classified by sensitivity, so confidential data stays on models inside your walls while routine work can use the best frontier AI.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.