I run a 60–90 minute, operations-led morning fire drill to prove whether an AI automation is production-ready. We inject realistic failures (input, partial data, downstream outages, rate spikes), use masked production-like data behind a feature flag, assign scripted roles, and apply numeric pass/fail thresholds to make a binary go/no-go call.
- A single 60–90 minute operations-run drill is the fastest way to validate an AI automation before production.
- Test the four failure classes that actually break automations: input errors, partial data, downstream outages, and rate/latency spikes.
- Set numeric thresholds you can measure in one session so the approver can make a binary go/no-go decision.
- Run the drill on masked production-adjacent data behind a feature flag and reversible toggles to avoid customer impact.
- If the drill fails, assign priority fixes and owners the same day and schedule a re-run within two working days.
Signs this is happening in your business
- Your exceptions queue balloons every Monday because the automation drops fields on oddly formatted PDFs.
- You get late-night alerts about degraded outputs that no one acknowledged in the morning.
- Approvals stall because the automation misreads documents or omits required fields.
- Third-party timeouts create partial updates that operations reconcile manually after the fact.
- Throughput spikes trigger throttling and a growing backlog that ops must clear.
- Users see inconsistent records depending on which system updated first.
- Developers keep patching edge cases with no clear owner or acceptance criteria.
If several of these are true at the same time, the problem is usually the process rather than the people running it.
What this is costing you a year
(Team hours per week x hourly cost of employees x 52) = annual payroll loss
- People doing this work4
- Hours each per week6
- Hourly cost of employees$45
- Weeks52
The annual payroll cost of four people each spending six hours a week on the manual work the automation should remove.
Find the number before you spend anything
Run the formula above on your three most repetitive processes. If any of them clears a year of a salary, that is the one to bring to a call.
Morning fire drill proves production resilience in 60–90 minutes
Run one 60–90 minute operations-led drill that simulates realistic production failures, assigns clear roles, uses masked production-like data behind a feature flag, and produces numeric pass/fail results. I run these drills to see whether an automation reduces payroll drag and operational risk rather than hiding the work behind silent failures.
Who runs the drill and what each role does
Make operations the owner: they run the clock, verify business outcomes, and sign off. Required roles are ops lead, on-call developer, subject-matter expert, approver and recorder. Give each person a one-page runbook, name a single substitute, and assign responsibilities before T=0 so the drill can measure detection and recovery cleanly.
Ops lead runbook (one page)
The ops lead opens the drill, keeps time, announces scenario starts and stops, runs the checklist, records timestamps for detection and recovery, and makes the go/no-go call. Before T=0 they confirm the feature flag and masked data are in place and that the recorder is ready to capture evidence.
- Pre-drill: confirm test data, feature flag, on-call dev and SME availability.
- T=0: announce scenario name and start time to all participants.
- Record timestamps for detection, escalation, mitigation and final verification.
- Make the pass/fail call for each scenario and sign the drill log.
On-call developer runbook (one page)
The on-call developer handles escalations and performs the minimal technical recovery that restores correct business outputs—rollback, toggle, or the quick fix specified in the runbook. They estimate a permanent fix, log the root cause, and record every change and timestamp in the drill record.
- Bring documented rollback steps and access to the feature flag control.
- If a quick fix exceeds the threshold, declare a containment action and hand off a ticket to engineering.
- Log every change and its timestamp in the drill record.
If you do not set numeric recovery thresholds before launch, your pilot will quietly become an undocumented manual process.
SME runbook (one page)
The SME validates outputs against business rules, compares results to the golden sample, and marks data quality pass or fail. They translate technical findings into business risk, advise required follow-up actions, and can veto a scenario if data integrity is compromised even when the system appears to be 'up'.
- Have the golden sample answers available to compare with automation outputs.
- Mark outputs as pass/fail and note discrepancies with examples.
- Advise on required follow-up actions and estimate operational impact.
Approver and recorder runbook (one page)
The approver, usually the operations manager, judges business risk and payroll impact and signs the go/no-go. The recorder captures timestamps, screenshots, exported records and the time-stamped chat transcript. Their work creates the clear evidence that supports the final decision and assigns owners for any failures.
- Recorder captures evidence: screenshots, exported records, and the time-stamped chat transcript.
- Approver signs off on pass/fail and assigns immediate owners for any fail items.
Six scenarios to simulate in the morning fire drill
Script a happy path plus five realistic failure classes: input errors, partial data, downstream outages, rate/latency spikes and human-approval delays. For each scenario write a short failure injection, the detection method, the containment action, and the measurable acceptance criteria you will use during the 60–90 minute session.
- Scenario 1 — Happy path: run a normal batch through masked production-like data and confirm outputs to baseline timing and format.
- Scenario 2 — Input error: feed an invoice with a corrupted field or unusual layout and confirm the automation flags it and routes it to a human queue.
- Scenario 3 — Partial data: send a record missing a mandatory identifier; confirm the automation flags it, does not write partial records, and creates a ticket for SME review.
- Scenario 4 — Downstream failure: simulate a third-party timeout or error; confirm the automation retries to policy, then halts and notifies the on-call developer if retries fail.
- Scenario 5 — Rate/latency spike: burst a batch that exceeds typical throughput; confirm the automation queues safely without data loss and exposes backlog metrics to ops.
- Scenario 6 — Human-approval delay: simulate an unavailable approver and confirm escalation to an alternate approver within the agreed SLA.
Identify your biggest operational bottleneck
Most owners already know which process is the problem. What they do not have is the annual figure attached to it. That is a 15-minute conversation, not a project.
Pass/fail criteria you can measure in one session
Set numeric thresholds you can measure in a single session: detection time, recovery time, test-slice error rate and data integrity. Require the test slice to meet every threshold before you call the drill a pass; do not base the decision on uptime alone.
- Detect within 10 minutes: the automation or monitoring must surface and log the abnormal condition within ten minutes of the injected fault.
- Recover within 30 minutes: a containment or recovery action restores correct business outputs within thirty minutes.
- Error rate under 5% for test records: of the test slice (50–200 masked records), fewer than 5% may be unrecoverable or mis-processed.
- Data integrity 99%: critical fields (IDs, amounts, names) must match the golden sample for 99% of test records.
- No irreversible writes: any action that would affect customer billing or external systems must be behind a reversible toggle during the drill.
Automations without a named owner fail noisily and waste attention and salary.
What test data and environment to use without breaking production
Run the drill on the same systems the automation will use but restricted to a bounded test slice so no real customers are affected. Use masked production-adjacent data, a feature flag that isolates writes, and reversible toggles or sandboxed third-party calls. Document the exact test boundary and who can flip the flag.
- Mask production data by replacing personally identifiable fields but keep structure and edge cases.
- Run the drill under a feature flag so outputs go into a parallel test queue, not the live queue.
- Route external calls to a sandbox or use call stubs; if you cannot sandbox, use a recorded replay with identical timings.
- Document exactly which systems are in the test boundary and who can flip the flag back if you see real impact.
60–90 minute runbook: minute-by-minute timeline
A tight timeline forces decisions and surfaces gaps. Stick to the clock, record every timestamp, and treat each scenario as time-boxed: prep, six scenarios, verification, and decision. Keep the session short so containment remains the priority and permanent fixes stay out of the window.
- T–15 minutes: Prep. Ops lead confirms participants, feature flag, masked data, and the recorder is ready.
- T=0: Kickoff. Ops lead states objectives, scenario order, and pass/fail thresholds.
- T=1–10: Scenario 1 (happy path). Baseline timing and outputs.
- T=11–25: Scenario 2 (input error). Inject malformed input. Record detection time and routing.
- T=26–40: Scenario 3 (partial data). Inject missing fields. SME verifies flagged records.
- T=41–55: Scenario 4 (downstream outage). Simulate third-party error. Developer performs retry policy or containment.
- T=56–70: Scenario 5 (rate/latency). Burst throughput. Ops watches backlog and queuing behavior.
- T=71–80: Scenario 6 (approval delay). Simulate approver unavailability and watch escalation.
- T=81–85: Verification. SME reviews outputs, recorder compiles evidence, ops lead confirms timestamps.
- T=86–90: Decision and sign-off. Approver signs pass/fail per scenario. If any fail, assign owners and schedule re-run.
How this commonly fails in practice and why
Drills usually fail for avoidable operational reasons: unclear ownership, missing sandboxes, or hidden irreversible writes. The predictable sequence is a fault appears, decision stalls, engineers start permanent fixes during the window, and containment is lost. I watch teams turning the drill into a bug-fixing session and losing measurable recovery times.
- Failure sequence: scenario exposes issue → no clear containment step → developer makes an unlogged change → data gets written to production → rollback becomes complex and time-consuming.
- A missing timestamp or an unlabeled detection makes the drill useless for measuring detection speed.
Record, prioritise, re-run and hand over
If any scenario fails, triage immediately: record the failure mode, severity, quick containment, permanent fix estimate, and owner. Create a permanent engineering ticket, hand ownership to operations with a clear escalation path, and schedule a re-run once containment is implemented—ideally within two working days.
- Use a simple severity scale: S1 (blocks go-live), S2 (degrades but contained), S3 (minor discrepancy).
- Assign immediate owners in the drill log and create a permanent ticket for engineering with the root cause and acceptance criteria.
- Re-run condition: the defect is contained and a rollback or flag is available; re-run within two working days.
- Handover checklist: runbooks stored in a single location, monitoring alerts mapped to owners, escalation contacts listed, and the feature flag documented.
- For ownership guidance see the automation ownership model. If you want an expert to convert a successful pilot, see convert automation pilot production.
Next step you should take tomorrow
Book 90 minutes on a Monday morning, assemble the ops lead, on-call dev, SME, approver and recorder, and run the scripted drill above using masked production-like data under a feature flag. Capture timestamps and make a binary go/no-go decision. If you want help designing the drill or converting a pilot, see how I can work with you and services.
Where businesses waste the most time
- Re-keying invoicesHours of manual typing, frequent transcription errors and delayed paymentsKeep your accounting package; add an extractor that places parsed invoices in a review queue rather than writing directlyA person still checks: A person spot-checks flagged invoices and approves uncertain parses
- Manual ticket triageBacklog and slow routing when staff are out, inconsistent prioritisationKeep your helpdesk; add a filter that routes only low-confidence cases to people and auto-closes clear onesA person still checks: An ops person reviews the queue of uncertain tickets
- Ad hoc monitoringAlerts go to group chat with no owner, leading to late detection and repeated escalationsKeep existing monitoring; map alerts to named owners and automated escalation rulesA person still checks: Named on-call person acknowledges and decides containment
How this compares with what you already run
| Operational area | The manual way | The automated way | Annual business impact |
|---|---|---|---|
| Ticket triage | People read every incoming ticket | System flags and routes uncertain tickets | Saves hundreds of staff hours per year |
| Invoice rekeying | Staff transcribe invoice fields into accounting | Automation places parsed fields for quick QA | Reduces repetitive errors and processing time |
| Approval bottlenecks | Approvals wait on a single manager | Automation escalates and routes to alternates after SLA | Fewer delayed approvals and faster cash flow |
| Monitoring and alerts | Ops reacts with ad hoc checks | Automation emits clear alerts with owner and severity | Less firefighting and faster incident response |
| Data reconciliation | End-of-day manual matching of records | Automation highlights exceptions for human follow-up | Lower reconciliation overhead across the year |
| Customer replies | Support crafts every reply individually | Automation drafts replies for human approval | Frees agent time and speeds response SLAs |
What the end-to-end workflow looks like
- Receive invoiceInvoice arrives in the intake system and is logged.Automation parses invoice fields and assigns confidence scores
- Confidence checkParsed data is evaluated against validation rules.High-confidence records advance automatically; low-confidence are queued
- Human reviewAn operations person reviews queued items and corrects fields.Staging queue shows suggested corrections; human edits if needed
- ApprovalManager confirms the processed invoice for payment.Stays human
- Post-processing writeFinalised invoice is written to the accounting system.Writes are behind a feature flag and can be toggled back
- ReconciliationEnd-of-day check matches processed invoices with payments.Automated reconciliation highlights mismatches for human follow-up
The metrics that tell you it worked
| Metric | What it tells you |
|---|---|
| Time to detect | Minutes from fault injection to when the abnormal condition is visible to ops; measures monitoring effectiveness. |
| Time to recover | Minutes from detection to containment or restoration of correct business outputs; measures operational readiness. |
| Test slice error rate | Percentage of test records mis-processed or unrecoverable during the drill; measures accuracy under failure. |
| Data integrity | Percentage of critical fields that match the golden sample after processing; measures trust in outputs. |
| Escalation latency | Minutes between an alert and an acknowledged escalation by a named owner; measures ownership clarity. |
| Manual touch time | Total human minutes spent on the test slice during the drill; measures residual operational cost. |
| Re-run interval | Time between initial failure and verified re-run; measures responsiveness and fix prioritisation. |
When this is the wrong move
- You do not have a feature flag or reversible toggle that isolates writes—do not run the drill on live writes.
- You lack a masked, production-like dataset or a safe sandbox for third-party calls.
- You have no named on-call developer or approver available within the drill window.
- The process being automated is still changing daily—freeze the workflow before testing.
- You cannot capture time-stamped evidence (screenshots, logs, exports) for each scenario.
The mistakes that cost the most
- No ownershipThe drill starts; nobody acknowledges the alert; developers make undocumented fixes; detection and recovery times are lost and the drill produces no actionable data.
- Testing on live writesA scenario writes to production because the feature flag was misconfigured; the team spends hours reverting changes and dealing with customer impact.
- Turning the drill into a bug-finding partyEngineers begin long-term fixes during the window instead of containment; the team cannot measure recovery time and fails to produce a go/no-go decision.
Frequently asked questions
Can I run this drill without a developer on call?
No. You need an on-call developer able to toggle feature flags or perform containment. Without a developer you cannot measure realistic recovery times or safely reverse risky actions; if you cannot provide one, restrict the drill to read-only scenarios until you can.
How many test records should I include in a drill?
Use a slice large enough to surface edge cases—typically 50–200 masked records. That gives enough breadth to test parsing, rate behaviour and approval flow while keeping the session short and reviewable.
What if a scenario fails the drill—do I delay go-live immediately?
If an S1 failure occurs, do not go-live. For S2 failures you may proceed with compensating controls and a committed fix schedule; the approver signs the decision and assigns owners for fixes.
How often should I repeat the morning fire drill?
Run a full drill before any production cutover. After go-live, schedule smaller drills monthly for high-risk automations and after any significant change. Re-run within two working days for any failed scenario until it passes.
Can the drill be used to train staff?
Yes. The drill doubles as operational training because it clarifies roles and escalation. Keep training separate from the pass/fail decision so learning does not blur measurement.
Stop paying salary for robotic work
Bring your messiest process to a 15-minute audit and we will map where the hours actually go, live on screen. You leave with the number whether or not you work with me.