I design safety prompts as production artifacts tied to named failure modes: system instructions, constraint lists, few‑shot examples and self‑checks. I version and unit‑test them with synthetic and adversarial scenarios, roll them out as small canaries under scoped access, and monitor overrides and risky outputs before widening permissions.
- Prompts only cut risk if you treat them like code: versioned, tested, and owned.
- Every prompt must map to a single, named failure mode and the business harm it prevents.
- Unit tests, synthetic adversarial cases and staged canaries make prompt changes auditable and reversible.
- Prompts reduce risk far more when paired with scoped permissions and human checkpoints.
- If you lack baseline metrics, design prompts that create them before you expand access.
Signs this is happening in your business
- One person reads every transaction because the agent keeps making mistakes.
- Staff routinely override agent outputs without recording why.
- Secrets, access keys or documents appear in outputs or external messages.
- You cannot find the exact prompt that produced a disputed decision.
- Rollouts replace old prompts with no fallback or rollback plan.
- Engineers edit prompts directly in production without tests or history.
- The agent re-submits the same failing action and forms an escalation loop.
- Support volume drops but chargebacks or corrections rise after deployment.
If several of these are true at the same time, the problem is usually the process rather than the people running it.
What this is costing you a year
(Team hours per week x hourly cost of employees x 52) = annual payroll loss
- People doing this work4
- Hours each per week6
- Hourly cost of employees$45
- Weeks52
The annual payroll cost of four people spending six hours a week each on manual work the automation could reduce.
Find the number before you spend anything
Run the formula above on your three most repetitive processes. If any of them clears a year of a salary, that is the one to bring to a call.
How I design safety prompts for automations
I design prompts by function and by the failure they must stop. I separate system instructions, constraint lists, few‑shot examples, adversarial appendices and self‑checks so each element has a clear role. That forces you to pick the minimal instruction that prevents the named harm.
- System instructions — define role, permitted actions and tone.
- Constraint lists — explicit forbiddens: exports, credentials, transfers.
- Few‑shot examples — show correct outputs for tricky cases.
- Adversarial appendix — record inputs that should be refused and the refusal text.
- Self‑checks — machine‑readable consistency checks the agent must run on its answer.
Map prompts to the failures that cost you money
I require a named failure mode for every prompt. If you cannot state which failure a prompt prevents and the business harm it avoids, deploy nothing. Start by listing top failures, write one sentence of business harm for each, then design a prompt element that directly prevents that harm.
- Hallucination → few‑shot examples plus a self‑check verifying known fields.
- Data exfiltration → constraint list forbidding transmission of sensitive fields or documents.
- Incorrect transactions → require a machine‑readable summary plus a human‑facing approval line.
- Privacy breach → include adversarial cases and a refusal rule when private data is referenced.
- Escalation loop → add an attempt counter and a rule to stop and escalate after N attempts.
Treat prompts like production: version, test, deploy
I manage prompts as production code: change history, automated tests, staging, and controlled rollouts. That discipline gives you rollback, traceability and the ability to tie a prompt version to an access decision. No owner, no history, no audit.
Treating prompts as informal text guarantees you cannot audit who caused a bad decision.
Unit tests, synthetic scenarios and canary rollouts
I write short deterministic unit tests that check structure and refusals for known inputs. I maintain a synthetic suite that includes common, edge and adversarial cases and run it on every change. I then expose a tiny slice of real traffic under restricted permissions while monitoring for issues.
- Include common, edge and adversarial cases in your synthetic suite.
- Automate unit tests to run on every prompt change.
- Run the new prompt only on a small portion of traffic with limited permissions while you monitor.
Acceptance criteria you can apply immediately
I require clear acceptance criteria before widening access: unit tests must pass, adversarial cases must be handled, and human override and risky‑output rates must sit below agreed thresholds during the holding period. Record those thresholds with the prompt version before any rollout.
- Unit tests: deterministic cases pass.
- Adversarial cases: zero unhandled refusal cases.
- Canary: predefined override and risky‑output thresholds met over the holding period.
When prompts are enough and when you must change the system
I use prompts to control behaviour and stop bad outputs. I change the system when the root cause is capability: missing data, broken business rules, or overly broad permissions. Do not rely on prompts to enforce absence of access; fix permissions or workflows instead.
- Use prompts for tone, output shape, confirmations and refusals.
- Change the system for access control, audit trails, irreversible financial controls and role enforcement.
The cheapest prompt is worthless if it grants the agent unfettered access to sensitive data.
How prompts interact with permission controls and checks
Prompts limit what the agent should do; permissions limit what it can do. I design prompts assuming minimal access and only grant additional permissions after prompts pass canary testing and an approval. Tie prompt versions to explicit access lists so you can revoke capability by rolling back.
- Bind prompt versions to access lists: version X can run with write access, older versions cannot.
- Limit the agent's visible fields to the minimum required.
- Require documented approval for any prompt that expands permissions.
Identify your biggest operational bottleneck
Most owners already know which process is the problem. What they do not have is the annual figure attached to it. That is a 15-minute conversation, not a project.
Monitor, audit and stop agents fast
I log the exact prompt text and version with each transaction, plus input, structured output and any human override. Monitor overrides and risky‑output detectors and alert owners quickly. Have a playbook to pause agents, rollback prompt versions, and notify compliance and impacted teams.
- Log the prompt text and version with each transaction.
- Track override rate, risky‑output detections and time to stop.
- Trigger an immediate pause on a spike in overrides or any high‑severity risky output.
Who should write and own prompts
I assign one accountable owner per prompt set from a cross‑functional pairing: operations writes the rules, product or compliance reviews policy fit, and an engineer implements versioning and tests. Write prompts in pairs so domain knowledge and deployment risks travel together.
- Author: operations drafts prompts tied to failure modes.
- Reviewer: product or compliance checks policy fit.
- Implementer: engineer wires prompts into staging and automated tests.
A pilot without a kill criterion never ends; it only gets quieter.
Handover checklist for go/no‑go decisions
I require a checklist in the prompt's version history before any production access. It forces proof that the prompt reduces measurable risk rather than shifting it. Keep test results, approval records and the rollback plan with the prompt.
- Named failure mode and documented business harm.
- Unit tests and synthetic scenario results attached.
- Adversarial cases tested and handled.
- Canary results and the holding period metrics.
- Permissions mapped and approved.
- Rollback plan and owner contact confirmed.
Where this fails in practice, and why
I see programmes fail when teams treat prompts as throwaway text: no versioning, no tests and no owner. Small edits happen in production, you lose auditability, and a crafted input quickly exposes gaps. The other failure is over‑trusting prompts for access control — prompts persuade but do not prevent capability breaches.
- Typical chain: a prompt is edited live → an edge case was untested → adversarial input leaks data → no version history to audit.
- Detection often comes from frontline staff or compliance depending on the symptom.
Decision framework: build, buy, or hire an expert
I recommend building if you have engineers and clear failure modes. Buy when you need audit trail and monitoring fast with limited delivery resource. Hire an expert when failure modes or regulatory risk are unclear and you need an initial safe production set designed and handed over.
- Build: you have engineering bandwidth and the automation is core.
- Buy: you need speed, standard integrations and an audit trail out of the box.
- Hire: you need expert design, testing and handover before full rollout.
What to do next
If you want a safety‑prompt checklist and a short risk review for your top automation, book a 30‑minute call so I can audit one automation end‑to‑end and give a go/no‑go. If you will run the work yourself, start by mapping your top failure modes, writing one constraint list and three synthetic tests for the highest‑priority automation.
- Option 1: Book a 30‑minute review to get a tailored checklist and risk notes — see How to work together.
- Option 2: If you are building in house, review what a consulting engagement covers at Services.
- Option 3: For custom builds, review delivery options at Custom build.
- See my background at About the author.
Where businesses waste the most time
- Manual approvalsHours of qualified staff checking routine requests and slow decisionsAgent drafts the action and presents a short human‑facing summary while keeping existing systems unchangedA person still checks: A person reviews exceptions and signs off on transfers
- Reading every ticketStaff read large volumes to catch agent mistakesAgent filters tickets and flags low‑confidence items for review without replacing the support systemA person still checks: A person checks only the flagged tickets
- Re-keying dataTime lost copying data between systems and introducing errorsAgent fills the target form automatically while the source system stays the sameA person still checks: A person spot‑checks entries and approves unusual cases
How this compares with what you already run
| Operational area | The manual way | The automated way | Annual business impact |
|---|---|---|---|
| Approvals | A person reads every request | Agent proposes actions; person approves exceptions | Cuts payroll on routine approvals |
| Invoice handling | Staff re-key and check line by line | Agent extracts and flags uncertain invoices | Reduces reconciliation time and late fees |
| Support triage | Staff reassign cases manually | Agent classifies with confidence and flags low confidence | Frees staff hours for higher-value work |
| Data exposure | People email files and share private fields | Agent refuses forbidden fields and logs attempts | Lowers compliance incident costs |
| Escalations | Repeated manual corrections | Agent stops after N attempts and hands to a named person | Cuts recurring correction and churn costs |
What the end-to-end workflow looks like
- TriggerA customer request or event starts the processAutomation
- Context assemblyRelevant information is collected and scoped to the minimum neededAutomation
- Prompted decisionThe agent receives the designed prompt and produces a structured recommendationAutomation
- Human approvalA person reviews low‑confidence or high‑risk recommendationsStays human
- Action executionApproved actions are executed with scoped permissionsAutomation
- Logging and auditThe exact prompt, version and decision are logged for reviewAutomation
- Monitoring and pauseAlerts trigger a pause and rollback if thresholds breachAutomation
The metrics that tell you it worked
| Metric | What it tells you |
|---|---|
| Override rate | Percent of agent outputs that a human changes; measures prompt accuracy and trust. |
| Risky output detections | Count of outputs that violate constraint lists or trigger adversarial rules; an early safety signal. |
| Mean time to stop | Time from a high‑severity alert to pausing the agent; measures operational reflexes. |
| Human review hours saved | Hours humans no longer spend on routine checks because the agent handles them. |
| Number of prompt versions | How many prompt iterations exist; indicates change discipline and auditability. |
| Canary failure rate | Percent of canary transactions that fail acceptance criteria during the holding period. |
| Escalation count | Number of cases routed to humans after N attempts; measures prompt limits and escalation rules. |
When this is the wrong move
- When the action is an irrevocable financial transfer that requires dual approvals and reporting — use system controls.
- When the decision requires legal judgement or bespoke negotiation — prompts can assist drafting but not replace counsel.
- When you have no record of failure modes or baseline metrics — build measurement first before delegating authority.
- When your permissions model is all‑or‑nothing — give agents the least access and avoid relying on prompts to limit exposure.
The mistakes that cost the most
- Editing prompts in productionSomeone tweaks the prompt live → an edge case was not covered → an adversarial input triggers a breach → no version record exists → investigations are slow and costly.
- Relying on prompts for access controlPrompts say 'do not export confidential data' → agent still has access to the field → a crafted input causes data leakage → a compliance incident follows.
- Skipping adversarial testsTeam runs only common cases → adversarial inputs are unseen → prompt passes canary but fails when probed in production.
- No owner for prompt setsPrompts drift as people change them → nobody monitors override rate → users lose trust and start ignoring the agent's suggestions.
Frequently asked questions
How detailed should a safety prompt be?
Make it as detailed as needed to stop the named failure mode and no more. Start with a short system instruction, add a constraint list for forbidden actions, include two clear examples, and add self‑checks if outputs must match a known field.
Can prompts prevent data leaks by themselves?
No. Prompts reduce accidental disclosures but cannot replace scoped permissions and access logging. Combine prompts with field restrictions and alerting to prevent real data leakage.
How do I test a prompt without risking production data?
Use synthetic scenarios and anonymised samples to mimic real inputs. Run those tests automatically on each prompt change and expose the prompt to a small, restricted slice of real traffic before widening access.
Who should see the prompt version history?
Operations, product, compliance and engineers should all have access. The version history is the audit trail you need to reconstruct decisions and assign accountability for changes.
When should I hire an expert instead of doing prompts in house?
Hire an expert if you have unclear failure modes, significant regulatory risk, or no engineering support for versioning and testing. An expert sets initial prompts, designs adversarial tests, and hands over a governed process.
How long should a canary hold before I expand access?
Hold a canary long enough to collect meaningful exposure under real load and to observe your predefined metrics. Define the metrics and thresholds before the canary; if they fail, roll back and fix before retrying.
What logs are mandatory for auditing a prompt?
Log the exact prompt text and version, the input used, the agent’s structured output, any human overrides, and the user who granted permissions. Those records let you reconstruct decisions for compliance and incident response.
Stop paying salary for robotic work
Bring your messiest process to a 15-minute audit and we will map where the hours actually go, live on screen. You leave with the number whether or not you work with me.