I run a pre-tested 0–48 hour incident playbook: detect and isolate affected processes, flip traffic to a pre-configured alternate provider or a local model, degrade non-essential features with flags, serve cached outputs for urgent requests, notify customers, and invoke contractual transition assistance.
- Assume any external model provider can be removed with little notice and design to switch quickly.
- A short, practiced 0–48 hour runbook turns panic into repeatable decisions.
- Engineering that makes failover cheap saves far more than the cost of lost customers and executive time.
- Contract clauses for data export and transition assistance buy operational breathing room.
- Regular, realistic drills reveal the integration gaps that break during a real block.
Signs this is happening in your business
- Sudden spike in model errors or timeouts while other systems remain healthy.
- Automations queue up because no model responses arrive and human review doubles.
- Customers complain about missing or wrong automated replies that used to arrive quickly.
- Provider console shows active keys but API calls return authorization failures.
- Billing or access pages show a paywall or account lock while your app still sends requests.
- A partner tells you their access was restricted and downstream automations stop producing outputs.
- Team members start copy-pasting previous responses or drafting replies by hand.
If several of these are true at the same time, the problem is usually the process rather than the people running it.
What this is costing you a year
(Team hours per week x hourly cost of employees x 52) = annual payroll loss
- People doing this work4
- Hours each per week6
- Hourly cost of employees$45
- Weeks52
The cost of four people spending six hours each per week on manual work across a year.
Find the number before you spend anything
Run the formula above on your three most repetitive processes. If any of them clears a year of a salary, that is the one to bring to a call.
what to do if llm provider blocks access
Run your 0–48 hour incident runbook now: isolate impacted automations, flip critical flows to a pre-configured fallback, disable non-essential features with flags, serve cached outputs for urgent cases, notify affected customers, and start contractual transition steps. Treat the event as a production outage, not a negotiation.
0–48 hour runbook: detect, triage, degrade, communicate, rollback
Follow a strict time-ordered runbook from incident start to resolution. Assign roles immediately, run checks to confirm provider-side failure, and stop ad hoc decisions. The runbook forces fast, reversible actions so you stop margin leakage and buy time to recover.
short-term production fallbacks you can switch to now
Recover most customer-facing flows within hours by using prepared fallbacks: route to an alternate hosted provider, run a smaller local model for essential tasks, serve cached answers for frequent queries, or turn off non-essential features. Choose the fallback that preserves the most critical user journeys.
- Swap to an alternate hosted provider already configured in your system.
- Run a smaller local model on managed hardware for immediate text generation or classification.
- Serve cached answers for frequent queries to keep latency low.
- Use simple rule-based logic for tasks that were previously model-driven.
- Feature-flag non-essential capabilities to conserve capacity.
how to make failing over actually feasible — architecture changes that pay back fast
Make failover cheap with four concrete changes: a model abstraction layer so you can swap providers with a setting, request caching for frequent prompts, ops-controlled feature flags, and observability at the provider call site so you alert on provider failures, not just application errors.
- Model abstraction: a single integration point where you can swap providers without touching business logic.
- Feature flags: allow ops to disable specific capabilities per customer or globally.
- Request caching: cache common requests with expiry and store outputs for cold-start protection.
- Observability: instrument the provider call so you alert on failures and auth issues.
- Data portability: schedule regular exports of prompts, outputs and user data in a standard format you own.
Vendors change commercial terms faster than you change org charts; entitlement can evaporate overnight.
which contractual protections actually matter
Demand notice periods, defined transition assistance and clear data export rights. SLAs are worthless unless they map to operational signals you already monitor; ask legal to translate your SLIs into contract terms that trigger concrete vendor obligations you can measure and use.
- Notice period: at least 30 days for planned changes and temporary access during billing disputes.
- Transition assistance: defined engineering hours to help export data or re-route traffic.
- Data export: machine-readable exports of prompts, outputs and traces on demand.
- Operational SLIs: contract definitions that match your monitoring.
- Termination for convenience with extended migration support and pro-rated refunds.
how to measure the trade-offs: quality, cost and latency when failing over
Decide what quality loss you will accept and what cost you will tolerate. Define measurable thresholds for quality, latency and 30-day cost of the fallback, then use those thresholds during drills to validate whether a fallback is business-safe.
- Define minimum acceptable quality using a small sample of customer conversations judged by human reviewers.
- Measure latency delta between primary and fallback and set a maximum acceptable delay for customer flows.
- Estimate the 30-day run-rate cost of the fallback to decide whether to keep it active longer.
testing and drills: practices that actually find the holes
Run drills that touch every component. Do a low-risk monthly smoke test and a full failover quarterly. Record timings, failures and human-hours. The first realistic failures will happen during drills, not the real outage, and fixing them there is far cheaper.
- Monthly smoke test: flip a low-traffic customer to the fallback and validate end-to-end outputs.
- Quarterly full failover: move a production shard to the fallback for a planned 24-hour window.
- Post-drill review: record human-hours spent, bugs found, and blocked owners.
Identify your biggest operational bottleneck
Most owners already know which process is the problem. What they do not have is the annual figure attached to it. That is a 15-minute conversation, not a project.
vendor-selection checklist: operational questions that matter
Ask operational questions that show how the provider treats customers under stress. Focus on exported data, defined transition support and operational availability expressed in your terms, not on marketing. Use these questions during procurement to avoid surprises later.
- Can you get an immediate, machine-readable export of all account-associated data?
- Do you offer formal transition assistance hours and at what cost?
- How do you define availability in operational terms we already monitor?
- What is your policy for blocking access or changing pricing mid-contract?
A runbook you haven't drilled is an excuse for chaos, not resilience.
how this fails in practice and why runbooks still break
Runbooks fail because teams skip the boring prep: fallback credentials aren't stored, cached outputs are stale, or support uses an out-of-date spreadsheet. The sequence and verification matter more than the theory—practise the full path until it works reliably.
- Fallback documented but credentials not stored in the production secrets store.
- Support relies on a canned-response spreadsheet that goes out of date.
- Contract export exists but the format is unusable without engineering work.
what to build now (practical backlog items you can finish in weeks)
If your team is small, do four things first: add a model abstraction, add ops-controlled feature flags for top customer features, enable request caching for frequent queries, and create a verified, automated export of model inputs and outputs. These are achievable in weeks.
- Abstraction point where providers are swapped with a single setting.
- Ops-controlled feature flags with per-customer overrides.
- Cache for top 100 frequent prompts with expiry.
- Automated weekly export of request and response logs in a standard file.
when you should hire help and what to ask them to do
Bring external help if you cannot complete the four essential backlog items in 30 days or lack senior technical ownership. Ask them to deliver a tested runbook, two independent failover paths, and a simple contract checklist you can use to demand transition assistance. See services for engagement options.
- Deliverable 1: a documented 0–48 hour runbook with role assignments and test history.
- Deliverable 2: two independent failover paths that can be engaged in under one hour.
- Deliverable 3: a simple contract checklist and amendment template to demand transition assistance.
- If you want to work directly, see how to work with me and as a tech consultant.
what to do next
Run a 30-minute resilience check tomorrow: pick a low-impact automation, flip it to the fallback for 30 minutes, and record the time and human-hours required. That single exercise will show whether your fallbacks are real or aspirational and what to fix first. See clients ecosystem for examples.
- Pick one routine automation with low customer impact.
- Flip it to your fallback for 30 minutes and measure results.
- If you cannot do that within an hour, schedule a resilience audit.
Sources and further reading
- 😺 Claude crossed a paywall on its own — Google News
Where businesses waste the most time
- Re-keying customer responsesHours per week spent copying historical answers into new ticketsStore and serve common responses from a shared cache so staff copy lessA person still checks: Approves exceptions and updates cache entries
- Manual escalation when model failsEngineers triage routine failures instead of fixing root causeCreate a fallback routing rule that sends failed cases to a review queue without replacing existing systemsA person still checks: Reviews the queue and re-trains rules
- Pulling data for vendor disputesLegal and ops spend days preparing exportsAutomate weekly exports of request and response logs into a standard file kept in your storageA person still checks: Verifies exported files and signs off on transfers
How this compares with what you already run
| Operational area | The manual way | The automated way | Annual business impact |
|---|---|---|---|
| Customer replies | Support writes answers manually | Automations generate replies with fallbacks | Hours of support saved |
| Invoice processing | Team rekeys invoices | Automated read and verification with exceptions | Fewer billing mistakes |
| Lead qualification | Sales sifts emails by hand | Automated triage with human review | More leads handled |
| Operations alerts | Engineers investigate each alert | Models pre-triage then escalate | Reduced time-to-fix |
| Regulatory export | Legal requests files manually | Automated exports on demand | Faster compliance |
What the end-to-end workflow looks like
- Incoming request receivedSystem captures the customer message and places it into the work queue.Automates work allocation to the processing routine
- Model generates a draft replyThe model returns a candidate reply used to inform the next step.Automates generation using the configured provider
- Quality checkA quick human or automated filter verifies the reply against safety rules.Automated checks with human escalation for edge cases
- Serve reply to customerApproved replies are sent and logged against the customer record.Automates delivery and logging
- Monitor failures and fallbackIf the model fails, the work is routed to the fallback path.Automates routing; fallback engages cached or alternate provider
- Post-incident reviewFailures are reviewed and permanent fixes scheduled.Stays human
The metrics that tell you it worked
| Metric | What it tells you |
|---|---|
| Failed-response rate | Share of model calls that return errors or timeouts; a rise signals provider trouble. |
| Human intervention minutes | Minutes per day that staff spend fixing automated replies; it measures hidden operational cost. |
| Fallback engagement time | Seconds from detection to the fallback being active; this measures runbook effectiveness. |
| Cache hit rate | Proportion of requests served from cache rather than the provider; higher rates reduce exposure. |
| Customer-visible incidents | Number of customers reporting degraded automation per week; it captures reputational impact. |
| Export readiness | Time to produce a validated data export on demand; this measures contractual readiness. |
When this is the wrong move
- You have no dependency on a provider for production work.
- Your automations are trivial and manual handling costs less than building fallbacks.
- You are mid-way through a planned major product rewrite where disruption is expected.
The mistakes that cost the most
- Failover not preconfiguredThe team documents a fallback but never stores credentials in production; when blocked they scramble to find keys and failover takes days.
- Relying on SLA aloneLegal points to an SLA but operations has already lost customer trust; SLA payments cannot restore that time.
- Skipping drillsThe runbook exists but no one has practiced flipping the flag; during the outage the flag path reveals missing migrations and breaks other systems.
Frequently asked questions
Can I rely on a provider's SLA to keep me safe from access blocks?
No. SLAs often exclude commercial or policy blocks. Translate your operational signals into contractual SLIs, insist on transition assistance and data export rights, and verify those clauses during procurement so you can act when access is restricted.
How fast can I realistically switch to an alternate provider?
If you preconfigure an alternate, you can switch critical flows within an hour. Without preconfiguration, expect days of engineering to adapt prompts, credentials and monitoring. Preparedness is what makes the difference.
What is an acceptable quality drop when failing over?
Acceptable quality loss is a business decision. Pick a small representative sample of customer interactions, rate fallbacks with human reviewers, and set a pass/fail threshold you use in drills to determine whether a fallback is business-safe.
What contractual clause gives me the most leverage if access is cut?
Transition assistance and data export clauses give immediate leverage. They require the provider to hand over machine-readable exports and provide hands-on help, which buys time to switch providers or run local models.
How often should we run failover drills?
Run a low-risk smoke test monthly and a full production failover quarterly. That cadence keeps the runbook current and reveals integration gaps before they become outages.
Stop paying salary for robotic work
Bring your messiest process to a 15-minute audit and we will map where the hours actually go, live on screen. You leave with the number whether or not you work with me.