A construction firm builds immutable backups after a near-miss, with object lock, a separate admin identity and a restore test of estimating and ERP data
A commercial construction firm of about 180 staff in South Florida, a leader in its segment.
- Scale
- About 180 staff, 6 servers (2 hosting the ERP and the estimating database), about 4.5 TB, 1 object-locked bucket, 30-day retention lock, 1 offsite copy, 3 restore tests in the first quarter
- Engagement
- Two weeks of assessment, about five weeks to build, then a restore test and a quarterly repeat
The challenge
A bid deadline week almost became a disaster. Malware reached a file server through a subcontractor's remote access, and the firm's backup server, which sat on the same network and used the same domain administrator account, started deleting restore points before the antivirus tool stopped the process. A person happened to notice the failed jobs and pulled the network cable. The firm lost nothing, but 11 of 30 restore points were gone and no one could say how long the backups would have lasted. The estimating database and the ERP held everything the company needed to bid, bill and pay, and no one had ever restored either from backup to see how long it would take.
Backups fail in two ways: they are missing when needed, or an attacker deletes them first. Object lock in compliance mode answers the second, since neither an administrator nor the provider's root account can remove a locked object before its retention date. The separate admin identity matters just as much, because a lock only helps if the same credential cannot also change the storage account.
A restore test is the only proof that a backup is a backup. Timing the return of the estimating and ERP data, and having the estimator and the controller check the totals, replaced a guess with numbers the firm's leadership and its insurer could rely on.
What our team delivered
- A review of how the backups could be destroyed: the backup server was joined to the domain, the backup repository was a share the server could delete, the console used a domain admin account without separate MFA, and the offsite copy was a cloud folder synced by the same account. Anyone who held that one credential could erase every copy
- Immutable copies: backups are written to a bucket with object lock in compliance mode and a 30-day retention period, so no one, including an administrator or the cloud provider's root account, can delete or shorten the lock before it expires. Versioning stays on and the lock covers the data as it is written, and a lifecycle rule removes copies only after their lock expires
- A separate admin identity: the backup system left the domain, uses its own accounts with phishing-resistant MFA, and the storage keys belong to a service identity that can write new backups but not delete or change the lock. The credentials for the storage account's owner are held by the owner of the firm and one other executive, stored offline, and every use triggers an alert
- Application-aware backups of the ERP and the estimating database (SQL Server) with transaction log backups every 15 minutes, so a restore can stop at a point just before an incident instead of at last night's copy, plus a copy of the document server with plans and bid packages
- A restore test into an isolated network: the estimating database, the ERP and the file shares were restored from the immutable copy, then opened by the estimator and the controller, who ran a real estimate and a trial payroll report and compared the totals with production. The test measured time to first usable data and time to full service
- A runbook with the order of recovery (identity, ERP, estimating, files), who decides to restore, how to prove that the backup is clean before restoring from it, and a quarterly repeat with a different system each time
Result
The first restore test took 9 hours to get the ERP usable and about 20 to bring back everything on the list, against a guess of 'a couple of hours' before. It found that the estimating database restored but its licence server did not, and that the ERP's file attachments sat in a share missing from the backup scope; both were fixed and the test repeated. Backup jobs now alert on any failed or deleted object, and a red-team style test of a stolen backup admin credential could write new backups but could not delete or shorten a lock. The firm's insurer accepted the object-lock design and the restore evidence at renewal. Immutability protects against deletion, not against backing up already corrupted data, so the team also checks restore points for malware before a recovery begins. This is an illustrative engagement that describes the kind of work our team does; details are composite.
Get this for your company
Related services with fixed prices and a written scope.
Emergency package: security engineer in 1 hour
Is your company in a cybersecurity crisis? Purchase this package and a security engineer will call you within 1 hour.
Callback within 1 hour, 24 hours a day, 365 days a year
$1,950 per incident
Insurance-ready bundle
The three things cyber insurers and clients ask for most: an external assessment, a phishing simulation for 25 users and a disaster recovery plan.
Assessment report in 5 business days; plan draft in 15 business days
$5,400 per bundle, phishing simulation for 25 users
Managed IT and security (MSP)
One monthly price per team member: help desk, device management, patching, backup monitoring and security basics.
Onboarding starts within 10 business days
$100 per team member, per month
Show 4 more
Initial assessment of public endpoints
What an attacker can see and reach from the internet: your domains, mail, websites, remote access and cloud exposure.
Report within 5 business days of kickoff
$1,000 per assessment
Cloud security review
A review of one Microsoft 365, Azure, AWS or Google Cloud tenant against recognized benchmarks.
Report within 7 business days of read-only access
$2,800 per tenant
Cloud management
We run your Microsoft 365, Azure, AWS or Google Cloud environment: access, cost, patching, backup and security settings. Priced per month for one tenant.
Onboarding within 15 business days
$1,900 per tenant, per month
Managed security monitoring
Endpoint detection and response watched by our analysts, priced per device.
Agents deployed within 7 business days
$18 per endpoint, per month
After you purchase, we contact you within 1 hour, any time of day or night, to start onboarding.
No payment is taken online. When we call you within the hour we confirm the scope and send an invoice (ACH, card or crypto).
Backup tiers: data lost versus time to recover
Recovery point objective (RPO) is how much recent data a restore loses; recovery time objective (RTO) is how long the restore takes. Faster tiers sit close to the production system and share its fate; slower tiers survive ransomware and site loss. The scale is logarithmic. Illustrative figures.
Email kept as write-once evidence
Each message is copied as a raw .eml with its headers and DKIM signature, hashed with SHA-256 and stored in an Amazon S3 bucket with Versioning and S3 Object Lock in compliance mode. An AWS KMS customer-managed key encrypts it, with key administrators separated from key users, and AWS CloudTrail logs every key use. A hash chain with a Merkle root receives an RFC 3161 timestamp, so a verifier can recompute the hash and check the timestamp and DKIM later. A delete attempt, even by the root user, is refused until the retention period ends.
The evidence pack and who asks for what
Twelve artefacts a firm can show about its AI use, and the party that most often asks for each: an insurer at renewal, an auditor or client questionnaire, or a regulator. A typical pattern only; the exact list depends on the policy, framework and sector.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.