Home / Guides / Tenant recovery drill
How to run a tenant recovery drill
Your backups pass their health checks. But have you ever actually recovered one customer from them, into live production, without disturbing the rest? A recovery drill is how you find out on a Tuesday afternoon instead of during an incident.
Most SaaS teams test that backups exist. Far fewer test that they can restore a single tenant from those backups — because in a pooled Postgres database, restoring one customer is an application-boundary problem your backup tooling was never designed to solve. A tenant recovery drill closes that gap: a rehearsed, read-only run of the exact steps a real recovery would take, ending in evidence that it works.
What an untested recovery actually costs
The failure isn't losing data — it's discovering, mid-incident, that recovering one customer means one of two bad options: roll back the whole database (and lose every other tenant's writes since the incident), or hand-write a fragile extraction script under pressure at 3am. A drill converts that unknown into a runbook.
What a good drill proves
- Completeness — the tenant graph resolves cleanly, with no ambiguous relationships blocking a real recovery.
- Coverage — object-storage files (uploads, invoices, exports) are mapped alongside database rows.
- Containment — a merge-back can be planned without risk to other tenants' data.
- Time — you know roughly how long a real recovery takes, so you can set an honest RTO.
The drill, step by step
1. Analyze (read-only)
Introspect the schema with a read-only role and build the tenant graph — classifying every table as root, tenant-owned, shared, ambiguous, or excluded. Nothing is written; only metadata leaves the database.
2. Report
Generate a readiness report that lists blockers: ambiguous relationships, missing keys, unsafe triggers, foreign-data-wrapper tables. These are the things that would stall a real recovery.
3. Extract and diff
Restore a snapshot to a scratch database, extract one representative tenant's closure to a bundle, and diff it against live data. This exercises the full path end to end without touching production.
4. Record evidence
Capture the result as a signed report: graph safety, row and object counts, blockers, and recommended manifest fixes. That artifact is your proof — for auditors, customers, and your own on-call runbook.
Drills stay read-only. Analyze, report, extract, and diff never write to production. Only an approved, hash-pinned restore or erase uses a write role — and that's a separate, deliberate step.
How often should you drill?
Schema drift is the enemy. Every migration can add a table, a relationship, or a cascade that changes the tenant graph. A practical cadence: an automated analyze/report job on a schedule that raises drift when the fresh graph no longer matches your last known-good manifest, plus a full extract-and-diff drill quarterly or after any major schema change.
Run your first drill
A TenantUndo Readiness Drill is a two-week, read-only recovery assessment for one app and one representative tenant — schema dependency map, object-storage coverage, a recovery runbook, and a signed report.
Book a recovery drill