I require live, scripted incident-response drills: a tabletop walkthrough, a timed alert-and-escalation drill, and a scoped canary or fault-injection test. Each drill must produce time-stamped artefacts, measurable MTTD/MTTR targets, runbook evidence, and contract clauses allowing immediate access revocation and remediation SLAs before production access.
- Slide decks and SOC letters are insufficient; require live, time-stamped drills.
- Run three tests: tabletop, timed alert-and-escalation, and a scoped canary.
- Set measurable MTTD, acknowledgement and MTTR thresholds as pass/fail gates.
- Collect runbook edits, ticket timestamps and immutable logs as procurement evidence.
- Convert failed drills into remediation plans, restricted access, or terminate negotiations.
Signs this is happening in your business
- Your team still reads every alert because vendor acknowledgements are absent.
- Vendor demands production credentials before showing logs or runbooks.
- Incident runbooks are a single-slide checklist or missing decision points.
- Engineers confirm alerts in chat without time-stamped evidence.
- Vendor uses subcontractors not listed in the contract.
- Critical automations can be switched off only by vendor staff with no local kill switch.
- Procurement accepted a SOC letter or slide instead of a live incident demo.
- Vendor calls remediation 'manual' and needs days of approvals.
- A near-miss ran automated steps on real customers during a change.
If several of these are true at the same time, the problem is usually the process rather than the people running it.
What this is costing you a year
(Team hours per week x hourly cost of employees x 52) = annual payroll loss
- People doing this work4
- Hours each per week6
- Hourly cost of employees$45
- Weeks52
The annual payroll cost of a four-person team spending six hours a week each on manual incident handling that an automation vendor should cover safely.
Find the number before you spend anything
Run the formula above on your three most repetitive processes. If any of them clears a year of a salary, that is the one to bring to a call.
How to test automation vendor incident response in three live drills
I run three live, scripted drills: a tabletop walkthrough to validate decisions and playbooks; a timed alert-and-escalation drill to measure acknowledgement and escalation windows; and a scoped canary or fault-injection test to prove logging, rollback and forensic capture. Each drill must yield time-stamped artefacts you can attach to procurement and contracts.
Treat the vendor as an engineering partner. Insist they produce evidence you collect directly: screenshots of alerts with timestamps, chat and ticket timestamps, runbook revisions made during the drill, and logs proving system state before and after remediation. Use those artefacts to decide pass, conditional pass, or fail and to draft contract clauses.
- Tabletop walkthrough: validates judgement and live runbook edits.
- Alert-and-escalation drill: measures acknowledgement, escalation and remediation starts.
- Scoped canary/fault-injection: proves automation behaviour, logging and rollback.
Pre-test checklist: the exact artefacts to demand before any drill
Require these documents in their operational form before any drill: an architecture diagram showing control points and approval gates; incident runbooks/playbooks that list decision steps and commands; alerting rules and sample alert text; logging and retention policy; current contact and escalation lists; identity and access model; subcontractor list; and a redacted sample audit log with timestamps.
Ask for the live versions, not sanitized examples. The architecture diagram must map data flows that touch your systems. Runbooks must specify decision points, required approvals, and exact actions the vendor will run so you can verify them during a drill.
- Architecture diagram with control points and fail-open/fail-closed notes.
- Incident runbooks for high-, medium-, and low-severity events with exact commands.
- List of alerts with triggers and the alert text you will receive.
- Logging policy showing what is logged, where and retention duration.
- Contact and escalation list with primary/secondary and time availability.
- Identity and access model: who can act and how access is revoked.
- Subcontractor and partner list with their access levels.
- Redacted sample audit logs showing a resolved incident with timestamps.
Three concrete tests with scripts you can run against any vendor
Use three runnable scripts: (1) a tabletop incident scenario walked through with named roles and recorded decisions; (2) a timed alert-and-escalation drill that injects a synthetic alert and enforces fixed acknowledgement windows; and (3) a restricted canary or simulated data-exfil exercise that captures forensic logs and triggers automatic rollback. Collect time-stamped evidence for each.
Below I provide step-by-step scripts, what to observe, and the exact artefacts to demand at the end of each test so procurement can verify the vendor’s operational capability.
Tabletop walkthrough script: show me the judgement, not slides
Run a 60–90 minute tabletop with named roles, a realistic incident scenario tied to your product, decision checkpoints, and a moderator who records choices and timestamps. Require the vendor to update the runbook during the session and deliver an immediately time-stamped version as evidence.
Script: pick a scenario that could affect your customers (for example, billing automation incorrectly credits accounts for 24 hours). Invite the vendor ops lead, their on-call, your product owner, security, and a moderator. Read the scenario and step through decision points; the vendor must state exactly which runbook step they will follow at each junction.
- Roles: vendor ops lead, vendor on-call, your product owner, your security lead, moderator.
- Duration: 60–90 minutes.
- Deliverables: annotated runbook updated live, timestamped meeting notes, follow-up actions with owners and deadlines.
Alert-and-escalation drill script: test response times under a clock
Inject a synthetic alert into the vendor’s monitoring channel and enforce strict windows for acknowledgement, escalation and remediation start. Record time-stamped screenshots, chat logs, ticket creation and system-state logs before and after remediation so procurement can verify MTTD, acknowledgement and MTTR against contract thresholds.
Script: agree a non-business-impacting synthetic alert (a flagged test message or metric crossing a test threshold). Announce the exact start time to a neutral observer who records timestamps. The vendor must acknowledge in their primary channel and create a ticket with a timestamp. If no acknowledgement within the window, the vendor must escalate automatically.
- Test start: announce time T0 and push the test alert.
- Acknowledgement window: acknowledge by T0 + 10 minutes.
- Escalation: escalate to secondary by T0 + 30 minutes if no primary acknowledgement.
- Remediation start: begin remediation by T0 + 60 minutes.
- Evidence: screenshot of alert, chat acknowledgements, ticket creation, pre/post logs.
Slide decks mask operational gaps that become outages and expensive manual work.
Safe canary / fault-injection script: prove the automation and rollback
Run a tightly scoped canary that exercises the exact automation path the vendor will use in production, with blast-radius controls, a pre-test snapshot of affected data, and an automated rollback trigger. Require full forensic logs that show each action and the rollback execution for procurement evidence.
Script: create a single labelled test customer or an isolated test queue that mirrors production behaviour. The vendor runs the automation against this canary. If the automation makes an irreversible change, the rollback must be automatic and verifiable. A neutral observer captures system state, execution logs, timestamps and rollback confirmation.
- Canary scope: one test customer or isolated data path only.
- Pre-test snapshot: vendor provides pre-test state capture.
- Rollback rule: automatic undo on failed checks.
- Evidence: action log with executor identity, rollback confirmation, post-test integrity check.
How to run tests safely and legally without causing outages
Limit blast radius, use non-production or labelled canaries, require written sign-off from legal, security and product, publish a notification window, and define rollback rules and a clear abort authority before the test starts. Record approvals and keep an independent audit trail so you can prove what happened.
Avoid touching live customer data unless legal has approved specific protections. When non-production isn’t available, a labelled canary path is the substitute. Define who can abort and how fast, and require the vendor to provide an immediate revocation mechanism for their access.
- Pre-test approvals: security, legal, product and an executive sponsor.
- Blast radius: single account or isolated flow only.
- Notification: scheduled window announced to impacted teams with abort contact.
- Rollback: automatic rollback conditions and manual abort procedure.
- Access kill switch: local ability to revoke vendor actions in under five minutes.
Measuring success: MTTD, MTTR and the evidence you should collect
Measure MTTD (time to detect), acknowledgement time, MTTR (time to remediate), runbook fidelity and log completeness. Require time-stamped artefacts for each metric and set clear pass/fail thresholds in procurement language so the tests translate into enforceable performance criteria.
Collect: time-stamped alert receipt, ticket and chat timestamps for acknowledgement and escalation, action logs for remediation steps, the runbook used during the incident (with live edits) and before/after system state. Those items are the only reliable proof of operational ability.
- MTTD: time from incident start to first alert received by vendor.
- Acknowledgement time: time from alert to explicit vendor acknowledgement.
- MTTR: time from remediation start to confirmed recovery.
- Runbook fidelity: proportion of steps followed exactly during the drill.
- Log completeness: full, time-stamped, tamper-evident logs that show actions.
Identify your biggest operational bottleneck
Most owners already know which process is the problem. What they do not have is the annual figure attached to it. That is a 15-minute conversation, not a project.
Numeric thresholds and pass/fail decision rules to use in procurement
Use firm thresholds as procurement gates: acknowledge alerts within 10 minutes, escalate within 30 minutes, start remediation within 60 minutes, and restore a canary within 120 minutes. Require two consecutive successful runs of each drill type to grant production credentials; a single failure requires a remediation plan and binding contractual penalties.
Those numbers are policies; adjust to your risk tolerance. If you need faster recovery, tighten the MTTR and require automated rollback or a local kill switch. Convert failures into conditional access: restrict the vendor to non-production paths until they pass twice in a row.
- Acknowledgement: 10 minutes.
- Escalation to secondary: 30 minutes.
- Remediation start: 60 minutes.
- Canary restore: 120 minutes.
- Acceptance: two consecutive successful drills.
What contractual protections to demand after tests
Translate test outcomes into enforceable clauses: specific breach notification timelines, remediation SLAs tied to MTTR thresholds, audit-log retention with tamper-evidence, immediate access revocation on failure, indemnities for data loss, and escrow of critical automation logic and runbooks.
Don’t accept vague phrases like 'reasonable cooperation'. Require concrete notification times, defined remediation steps if thresholds are missed, rights to on-demand audit logs, and a clause allowing immediate credential revocation during live incidents when SLAs are missed.
- Breach notification timeline and escalation obligations.
- Remediation SLAs tied to test thresholds with financial penalties.
- Audit-log retention and tamper-evidence (hashing or immutable storage).
- Immediate access revocation without multi-day notice.
- Escrow of automation code and runbooks with update cadence.
If a vendor cannot run a safe canary against their automation, do not give them production access.
Where this fails in practice and the sequence that kills deals
The common failure sequence: vendor delays or refuses live tests, supplies sanitized artefacts, runs a drill without proper blast-radius controls, produces no time-stamped evidence, and then offers runtime explanations. That pattern should end the deal or convert it into a tightly restricted pilot while you control access.
Refusal to run a live drill is the single biggest red flag. Insist on real timestamps and subcontractor transparency; accepting a one-off successful drill without repeatability is dangerous.
- Refusal to run a live drill is a red flag.
- Sanitised logs look like marketing; insist on real timestamps.
- Undisclosed subcontractors hide operational risk.
- One-off drills without repeatability are insufficient.
Red flags that should stop a deal immediately
Walk away or lock the vendor to non-production if they refuse live tests, have no runbooks, cannot show time-stamped logs, hide subcontractors, demand unrestricted production access, or lack a kill switch and rollback procedures. Treat these as binary procurement gates: either evidence and controls exist, or you deny production access.
A vendor that treats incident response as a sales conversation rather than an engineering discipline will cost you time and money later. Make the drills and artefacts mandatory.
- Refusal to run any live drill.
- No runbooks or only one-line playbooks.
- No time-stamped or tamper-evident logs.
- Undisclosed subcontractors with production access.
- Demand for unrestricted admin access with no local kill switch.
What to do next: the single action you can take tomorrow
Schedule a 90-minute vendor-validation session and demand the pre-test artefacts listed above; use the three drills here as your agenda, record everything, and make passing the drills a contractual precondition for production credentials. Attach time-stamped evidence to procurement files.
If you want external help, I run vendor-validation engagements that moderate table-top sessions, observe live drills and produce the audit package and contract language. See Services or book via [/how-can-i-work-with-you/].
- Step 1: Request the pre-test checklist documents from the vendor.
- Step 2: Schedule the tabletop and two live drills within 7–14 days.
- Step 3: Record and attach all artefacts to your procurement file.
Why many teams skip these tests and pay for it later
Teams skip tests because procurement accepts paperwork, engineers are busy, and vendors promise they will prove things in production. The result is longer outages, unexpected manual work, and executive time spent fixing preventable failures.
Automation hides work. If you don't force the vendor to show the work, your team will inherit it when the automation fails. Live proof is the only reliable antidote.
How I work and what a validation engagement covers
I run the drills, collect the time-stamped evidence, map failures into procurement clauses, and produce a short remediation plan tied to contract language. I moderate tabletop sessions, observe live drills, and deliver an audit package you can attach to vendor contracts.
If you need a facilitator to push the vendor for evidence and translate failures into enforceable clauses, see Services, the work in Portfolio and case studies, and About me.
Where businesses waste the most time
- Reading every alertWastes skilled time and creates alert fatigueFilter alerts to actionable ones and require vendor acknowledgement rather than human reviewA person still checks: A person reviews exceptions and unusual cases
- Manual remediation approvalsDelays fixes while waiting for sign-offPre-authorise vendor runbook steps for well-defined incidents with clear limitsA person still checks: A manager reviews out-of-scope cases
- Searching for logs after an incidentLong investigations and poor recovery notesRequire the vendor to keep time-stamped logs in an agreed place before accessA person still checks: A lead validates the forensic timeline
How this compares with what you already run
| Operational area | The manual way | The automated way | Annual business impact |
|---|---|---|---|
| Alert handling | Team reads every alert and triages manually | Vendor detects, acknowledges and follows runbook | Fewer hours on triage; fewer missed incidents |
| Escalation | Escalation is ad hoc and by chat | Timed escalation with documented handover | Faster resolution; less executive time chasing fixes |
| Remediation | Manual fixes and approvals take days | Automated rollback and verified restore for canaries | Lower downtime cost and fewer customer credits |
| Audit & logs | Logs are scattered and mutable | Immutable, time-stamped logs retained per policy | Faster root cause analysis and regulatory readiness |
What the end-to-end workflow looks like
- DetectionVendor monitoring triggers an alert when a threshold is crossed.Automation detects and sends the first alert
- AcknowledgementVendor acknowledges the alert in a time-boxed window.Automated ticket creation and notification
- TriageOn-call follows the runbook to classify severity and next steps.Automation gathers initial diagnostics; human confirms
- RemediationVendor executes remediation steps or triggers rollback.Automation runs pre-authorised remediation
- VerificationSystem state and canary checks confirm recovery.Automated checks validate state; human approves release
- Post-incident reviewVendor and your team review actions, update runbooks and log lessons.Stays human
The metrics that tell you it worked
| Metric | What it tells you |
|---|---|
| MTTD (Mean Time to Detect) | Time from incident start to the alert appearing in the vendor’s system; measures monitoring coverage. |
| Acknowledgement time | Time from alert to first explicit vendor acknowledgement; measures alert handling speed. |
| MTTR (Mean Time to Remediate) | Time from remediation start to confirmed recovery; measures remediation effectiveness. |
| Runbook fidelity | Proportion of runbook steps followed exactly during a drill; measures playbook accuracy. |
| Log completeness | Presence of full time-stamped logs for the incident, including actions taken; measures forensic readiness. |
| Escalation adherence | Whether escalation windows were honoured and the correct people notified; measures chain-of-command reliability. |
| Canary restore time | Time to restore an isolated canary to its pre-test state; measures rollback capability. |
When this is the wrong move
- When the vendor cannot or will not provide the pre-test artefacts and sample logs.
- When legal or compliance forbids even isolated canaries and there is no non-production path.
- When you lack internal owners for incident response and cannot act on remediation.
- When the vendor has uncontrolled subcontractors with unknown access.
- When the automation will touch highly sensitive data and you have not established data-handling controls.
The mistakes that cost the most
- Accepting slides as proofProcurement accepts slide decks and summaries; months later a real incident shows missing steps and long manual approvals because there was no live evidence.
- Skipping legal and security sign-offTeams run a quick test without legal approval, expose customer data, then face compliance fallout and remediation costs.
- Running a drill with broad blast radiusA poorly scoped canary touches customer records; rollback is slow and the vendor’s access model lacks a kill switch, causing outages and credits.
Frequently asked questions
Can vendors refuse to run live tests?
No. Require them as a condition for production access. If a vendor refuses, treat it as high risk: demand an explanation, insist on an alternative controlled test that produces the same artefacts, or restrict them to non-production access only.
What if we have no non-production environment?
Use a labelled canary that mimics production but affects only a single account or isolated flow. Get legal and security sign-off, limit the blast radius, and require automatic rollback controls before running the test.
How many drills are enough?
Require two consecutive successful runs of each drill type before granting production credentials. One pass shows capability; two shows repeatability and process maturity.
What if a vendor passes drills but fails later?
Enforce contractual protections from the tests: remediation SLAs, penalties, and immediate access revocation. Tests and clauses reduce but do not eliminate risk—keep human oversight in the loop and audit logs on demand.
Should I accept SOC reports instead of live drills?
No. SOC reports document controls but they do not prove the vendor can detect, escalate and remediate incidents against your automation. Live drills produce the operational evidence procurement needs.
Stop paying salary for robotic work
Bring your messiest process to a 15-minute audit and we will map where the hours actually go, live on screen. You leave with the number whether or not you work with me.