I treat the 'agent' label as marketing. Before you buy, require reproducible test cases, replayable logs that show inputs and steps, output provenance, RBAC on connectors, containment for third‑party actions, monitoring with alerts, correctness SLAs, and an exportable configuration and runbook — then run a 30–90 minute adversarial demo.
- Labeling a product an 'agent' is marketing; demand reproducibility and observability as the capability you buy.
- A 30–90 minute adversarial demo exposes whether the vendor can produce deterministic behaviour and safe containment.
- If you cannot export the agent configuration, tests and logs, you do not own the automation; the vendor does.
- Operational guarantees beat model brand names: monitoring, alerting, error recovery and runbooks prevent surprise payroll cost.
- Translate vendor usage into business cost per automated hour before you sign so you budget review and retries.
- Single‑person tribal knowledge is the common long-term cost; require role separation and change control up front.
Signs this is happening in your business
- Your ops team inspects every automated decision because they do not trust outputs.
- A vendor demo produces great answers live but cannot reproduce the same outputs twice.
- Connectors fail intermittently and the vendor issues patches with ad hoc invoices.
- You have only a final summary, not logs tying which data produced each decision.
- Billing jumps after launch because human review time and retries weren’t counted.
- One subject matter expert is the only person who knows how the agent was tuned.
- Compliance asks for provenance and the vendor cannot show source data or model version.
- The vendor refuses to export configuration, citing proprietary IP.
If several of these are true at the same time, the problem is usually the process rather than the people running it.
What this is costing you a year
(Team hours per week x hourly cost of employees x 52) = annual payroll loss
- People doing this work4
- Hours each per week6
- Hourly cost of employees$45
- Weeks52
The annual payroll cost of four people spending six hours each week on repetitive work that could be automated.
Find the number before you spend anything
Run the formula above on your three most repetitive processes. If any of them clears a year of a salary, that is the one to bring to a call.
How to evaluate vendor ai agent claims before buying
When a vendor shows you an 'agent' demo, don’t sign because the demo was clever. I require deterministic test cases, replayable logs with inputs and intermediate steps, output provenance tied to model version, RBAC on connectors, and an exportable configuration and runbook. Those are the features you must own, not a label.
- Deterministic test cases the vendor can run repeatedly and export.
- Replayable logs showing inputs, intermediate steps, and final outputs.
- Provenance for every output: which data, which model version, who triggered it.
- Role-based access control for connectors and credentials.
- Exportable configuration, prompts, and test suite at contract end.
Run this 30–90 minute practical test during any demo
Run a short adversarial test you can repeat in 30–90 minutes. I use baseline inputs, edge cases, adversarial prompts and a replay check because this sequence exposes reproducibility, error handling and containment failures quickly. Expect at least one failure; how the vendor responds tells you if the product is usable.
- Baseline (10–15 minutes): Provide 6 realistic inputs the agent should handle. Ask the vendor to run them and export the exact run logs and outputs. Acceptance: You can re-run the same inputs and get the same outputs.
- Edge cases (10–20 minutes): Include two malformed or incomplete inputs, one large input, and one rare but real case. Acceptance: The agent either handles them correctly or fails with a clear, auditable error and no side effects.
- Adversarial prompts (10–20 minutes): Ask for outputs that require refusing or escalating (e.g., requests to access customer PII or to perform irreversible actions). Acceptance: The agent refuses or escalates according to a visible policy.
- Containment and connectors (5–15 minutes): Watch an attempted connector action that should be blocked. Acceptance: The action is blocked and the attempt appears in logs with the reason.
- Replay and provenance (10 minutes): Request a replay of one run that shows each decision step, the model version used, and the source data. Acceptance: You receive a replayable log and can map outputs back to inputs and model version.
A shiny agent demo that won’t run its own tests on demand is a sales pitch, not a product.
What architecture questions force clarity
Ask where the model runs, how data flows, who holds credentials, and who can change prompts or policies. I insist on operational answers you can audit; marketing language is a red flag. Demand plain descriptions of storage locations, connector authorization, credential ownership and what changes require engineering.
- Where does the model compute happen: in the vendor’s environment, in your environment, or a hybrid? Explain in simple terms.
- How does data travel from your system to the agent and back? Ask for a step-by-step description you can map to compliance controls.
- Who stores and manages the credentials used for connectors? Demand least‑privilege handling and rotation policies.
- How are prompt changes, policy adjustments, and connector settings controlled? Ask for role separation and an approval trail.
- How are model versions managed? Require the vendor to record model version with every output and to offer rollback.
Operational guarantees you must demand before contracting
Put monitoring, alerting, correctness SLAs, error handling rules and incident postmortems into the contract. I insist on auditable guarantees: continuous recording of decisions, automatic escalation for risky outputs, defined remedies for SLA breaches, and documented retries and rollback procedures so you don’t buy unpredictability that eats payroll.
- Monitoring: continuous recording of decisions, error rates, and which model version produced each decision.
- Alerting: immediate alerts to your ops team for failures, escalations, or security-sensitive requests.
- SLA for uptime and for correctness in defined test cases; include remedies for breaches.
- Error handling: documented retries, rollbacks, and human‑in‑the‑loop escalation paths.
- Incident postmortems: vendor provides a written report for every serious outage and a remediation plan.
If you cannot export the configuration and test suite, you do not own the automation and you will pay more later.
Security and compliance checks that catch real risks
Treat data exfiltration as the primary risk and demand verifiable proofs. I refuse vendor-only assurances: require live demonstrations of blocked access, full audit logs, and isolation for subprocesses that call external services. Test PII detection and redaction and confirm who controls credentials and rotation policies.
- Data exfiltration test: inject a dummy secret into an input and watch for any attempt to send it out. If the vendor won’t run this, consider that a red flag.
- PII handling: vendor must show how the agent identifies and either redacts or escalates PII requests.
- Audit logs: every access, decision, connector call and policy change must be logged and exportable.
- Credential management: credentials must be stored under your control or under vendor control with transparent rotation and least‑privilege roles.
- Third‑party subprocess isolation: any call to external services must be contained and auditable; the vendor must show isolation mechanisms.
Translate vendor pricing and usage into business cost
Price the automation as a business line item. I add vendor unit costs to measured human review hours, retries and incident handling to calculate a real monthly operating figure. Do this before you sign so you budget staff time, connector maintenance and professional services into total cost of ownership.
- Measure: how many runs per month, average run time, and average human review minutes per run. Multiply by vendor unit cost and your review hourly rate.
- Include hidden costs: failed runs that require manual fixes, connector downtime that causes business delays, and vendor professional services for changes.
- Negotiate a clear unit of work price or a cap on monthly usage; insist on transparency around what triggers extra charges.
- Require exportability of configurations, test suites and logs so you can take the automation in‑house or to a new vendor.
Most operational costs appear after launch: monitoring, exceptions and single‑person tribal knowledge drive ongoing spend.
Identify your biggest operational bottleneck
Most owners already know which process is the problem. What they do not have is the annual figure attached to it. That is a 15-minute conversation, not a project.
Why these agents fail in practice and the failure sequence I see
Failure follows a sequence I have seen repeatedly: a rushed pilot with no exit criteria, accumulating exceptions, vendor fixes charged ad hoc, and a single expert who holds tribal knowledge. I stop projects that lack kill criteria; otherwise the pilot becomes a brittle, costly operational system.
- Pilot launches with vendor support and no exit criteria.
- Routine exceptions grow; vendor charges for fixes; no one updates tests.
- A subject matter expert becomes the only person who understands tweaks.
- Patchwork fixes lead to inconsistent outputs and compliance exposure.
- At renewal, surprise operating costs are revealed and the contract becomes toxic.
Short actionable buyer checklist you can use in procurement
Require these items in writing before you sign. I use this checklist in procurement: if a vendor resists any single line, treat that as a negotiation point. These items separate products you can operate from demonstrations you cannot.
- Deterministic test suite and the right to run it on demand.
- Exportable run logs and replay capability for every decision.
- Provenance: model version and source data recorded per output.
- RBAC for connectors and credential ownership clearly defined.
- Containment rules and a tested refusal/escalation policy for risky requests.
- Monitoring dashboards and alerts delivered to your ops team.
- Export of full configuration, prompts and tests at contract end.
What you should do next — a one‑day action plan
Schedule a 60‑minute vendor session and run the 30–90 minute practical test. I require the vendor to export logs and configuration at session end and to document any failures in writing. If you want help validating vendor claims, book a short scoping call via How to work together.
- Book a 60‑minute demo and send the practical test script in advance.
- Run the replay and containment checks; gather exported logs.
- Translate the vendor’s usage into monthly cost including staff review.
- Confirm export rights for config and tests in the contract draft.
- If you need technical leadership to assess the answers, consider a fractional CTO via Fractional CTO.
How these engagements typically get structured with outside help
When I advise, expect a short procurement phase, a pilot with strict acceptance criteria, and a handover package. For build work I separate advisory from delivery: advisory produces the test script and acceptance criteria; build delivers the automation, tests it, and hands over documentation and runbooks to your ops team.
- Advisory: procurement checklist, vendor test script, risk assessment. See Tech consulting.
- Build: custom automation, tests, and handover to your ops team. See Build tech for you.
- References and examples of prior work are listed in Portfolio and case studies.
- Background on why I take this approach is on About the author.
Where businesses waste the most time
- Re-keying customer dataWastes staff hours and introduces transcription errors that require later fixesHave the agent extract fields and append them to the existing system rather than replacing itA person still checks: Person checks flagged mismatches and approves final record
- Manual email triageTeam spends hours deciding who owns messages and what to doAgent attaches suggested owners and templates to emails while leaving your mailbox intactA person still checks: Ops lead reviews escalated or ambiguous messages
- Investigating failed paymentsMultiple people chase the same issue and duplicate effortAgent collects transaction evidence and creates a single investigation item in your workflow toolA person still checks: Finance person completes the resolution and records the outcome
How this compares with what you already run
| Operational area | The manual way | The automated way | Annual business impact |
|---|---|---|---|
| Customer onboarding | Staff copy and verify by hand | Agent fills fields and flags exceptions | Significant payroll savings annually |
| Invoice processing | Team reads invoices and posts payments | Agent reads invoices and escalates mismatches | Fewer late payments yearly |
| Support triage | Staff route tickets manually | Agent pre-sorts tickets with context | Faster responses across the year |
| Order exceptions | People investigate and chase suppliers | Agent gathers evidence and creates single ticket | Reduced repeat investigations annually |
| Contract review | Law reads clauses manually | Agent highlights risky clauses for review | Lower legal hours per year |
What the end-to-end workflow looks like
- TriggerA new event arrives that could start the automation, such as a new order or support ticket.Agent checks the incoming data and decides if it matches automated criteria.
- Pre-checkThe agent validates required fields and looks for obvious errors.Automates basic validation and categorisation.
- Data enrichmentThe agent gathers related information from known systems to build context for a decision.Agent performs enrichment while logging all sources used.
- DecisionThe agent applies rules and decides to complete, escalate, or refuse.Agent makes the decision and records the model version and rationale.
- Action executionThe agent performs permitted actions, such as updating records or creating tickets.Performs actions under role controls; blocked actions are refused and logged.
- Human reviewAn operations person checks exceptions and edge cases flagged by the agent.Stays human
- MonitoringThe system records outcomes, errors and changes for later analysis.Automated logging and alerting to ops team.
- PostmortemThe team reviews incidents and updates tests, prompts, and rules.Human-led updates with vendor support as required.
The metrics that tell you it worked
| Metric | What it tells you |
|---|---|
| Hours saved per week | Measures staff hours no longer spent on the task; translates directly to payroll savings. |
| Automation success rate | Percentage of runs that complete without human intervention; tracks automation reliability. |
| Human review hours per week | Total time staff spend checking agent outputs; indicates operational load after automation. |
| Time to detect failures | How quickly ops are alerted when the agent misbehaves; shorter times reduce business impact. |
| False positive rate | Rate of incorrect actions the agent attempted that required rollback or manual correction. |
| Cost per automated hour | Total vendor and staff cost divided by hours automated; helps compare automation to payroll. |
| Exceptions per month | Number of cases the agent cannot handle and routes to humans; a key driver of ongoing cost. |
When this is the wrong move
- When the volume is tiny and human review costs less than the vendor’s minimum monthly charge
- When decisions require bespoke expert judgment every time
- When you cannot export configurations or logs and must retain compliance control
- When regulation forbids automated decisioning for the use case
- When your data quality is so poor that automation would only amplify errors
The mistakes that cost the most
- Rushed pilot with no stop conditionLaunch a pilot without defining acceptance or kill criteria. Exceptions accumulate unnoticed, and the pilot quietly becomes operational without proven value.
- Trusting marketing for containmentAccept vendor assurance that the agent 'won’t do X' without an auditable test. Later a connector misfire causes unintended actions.
- No export or handover rightsYou rely on the vendor for everything. When the contract ends you lose operational capability and face reintegration costs.
- Ignoring runbook and observabilityNo monitoring or runbook is created. When the agent fails, your ops team wastes time tracing what happened and who changed prompts.
- Underbudgeting human reviewCount only vendor fees and forget ongoing review and exception handling. Payroll for review grows and erodes promised savings.
- Single‑person tribal knowledgeOne expert becomes the only person who can tune the agent. They leave and the automation becomes brittle and costly to maintain.
Frequently asked questions
Can I trust an agent demo that answers everything correctly in 10 minutes?
No. A quick demo is insufficient. Insist on reproducible test runs, exported logs and a replay that shows how each decision was made; the demo must be repeatable by you with the same results.
What is the single most revealing test I can ask for?
Ask for a replayable log that ties each output to input data and the model version. If the vendor can’t produce that, they cannot guarantee reproducibility or compliance.
How do I convert vendor usage into a monthly cost?
Measure runs per month, average human review minutes per run, and vendor unit price per run; multiply and add staff costs. That gives a monthly operating cost you can compare to payroll saved.
What contract terms protect me if the agent breaks after launch?
Require SLAs for correctness on your test cases, documented incident postmortems, a clear change control process, and export rights for configs and tests at contract end.
Can I keep the agent but move it in‑house later?
Only if the vendor agrees to export full configuration, tests and logs in a usable format. Insist this be contractually guaranteed and tested during procurement.
How much human review should I expect after automation?
Expect a material human monitoring load at first while exceptions settle. Measure human review hours per week and reduce that target over time by improving tests and rules.
What if the vendor refuses to run an adversarial test?
Treat refusal as a significant red flag. Either the product cannot safely handle risky inputs or the vendor cannot demonstrate containment — both unacceptable for production.
Do I need an outside consultant to evaluate vendors?
You can run the practical tests yourself, but an independent advisor speeds up technical translation, writes the acceptance criteria, and protects you from vendor spin. If you need this, see How to work together.
Stop paying salary for robotic work
Bring your messiest process to a 15-minute audit and we will map where the hours actually go, live on screen. You leave with the number whether or not you work with me.