A manufacturer's on-premises Exchange restored after a failed cumulative update, then made redundant
A manufacturer of about 900 employees with plants in three U.S. states.
- Scale
- ~900 mailboxes, one Exchange server before, a two-site DAG after
- Engagement
- 2 days of emergency recovery, then a 10-week redundancy build
The challenge
A cumulative update failed halfway on the company's only Exchange server and left services unable to start. Mail for plants and suppliers stopped, the last tested backup was months old and the server had no second copy of its databases.
What our team delivered
- Triage of the failed setup from its logs: which services, which Active Directory objects and which databases were affected, before anything was changed
- Recovery of the mailbox databases from the dirty-shutdown state, with the databases copied aside before repair
- Mail routing restored to users and plants through a temporary path while the server was rebuilt, so suppliers' mail was queued, not lost
- A rebuilt server on a supported version with certificates, connectors and the hybrid relationship with Microsoft 365 verified end to end
- A second Exchange server at a second site, a database availability group with a copy of every database, and a file share witness placed in a third location
- A patching calendar: updates staged on a test server first, each update applied one DAG member at a time, and backups and restore tests recorded
Result
Mail was back within the first two days and no mailbox data was lost. The company now loses a server or a site without losing mail access, and updates no longer need an outage window.
Mail that keeps flowing
Incoming mail is authenticated and scanned before it reaches anyone, mailboxes are copied across three regions so a server failure goes unnoticed, and outgoing mail leaves signed and encrypted.
Failover in under a second
ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.
Backup tiers: data lost versus time to recover
Recovery point objective (RPO) is how much recent data a restore loses; recovery time objective (RTO) is how long the restore takes. Faster tiers sit close to the production system and share its fate; slower tiers survive ransomware and site loss. The scale is logarithmic. Illustrative figures.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.