Cut LLM Token Costs in Automations Without a Rebuild

Signs this is happening in your business

What this is costing you a year

Reduce LLM token costs in automations quickly with routing and caching

I start with routing, short memoisation, and a single batching window because they remove the bulk of repeat token use fast and are reversible. Implement these behind feature flags, measure the impact on tokens and cost, and only then consider prompt compression or model swaps for remaining spend.

Begin by measuring tokens per call, calls per flow, and the model mix to prioritise correctly.

• Measure tokens per call and calls per flow for one week.
• Map which team or process triggered each call.
• Identify flows with high volume, repeated context, or many near-duplicate calls.

How to find where tokens are being spent and estimate dollars per process

I use a one-line formula to turn token behaviour into dollars per flow: tokens per call × calls per month × vendor price per 1,000 tokens. That converts noisy data into an ROI for each intervention and tells you where to focus engineering effort.

Do this measurement before you change prompts or models; it prevents wasted work.

• Formula: (tokens per call) × (calls per month) × (price per 1,000 tokens) = monthly cost for the flow.
• To estimate per-process annual impact: multiply the monthly cost by 12 and add any expected growth factor.

Diagnostic checklist: what to measure first

I collect five metrics in a representative week to decide which flows to optimise: tokens per request, requests per flow, model used per call, fraction of repeated context, and human review hours tied to that flow. Those numbers give finance the evidence they need.

Extrapolate weekly counts to monthly totals only after a clean sample period.

• Tokens per request (average and 95th percentile).
• Requests per flow per day and monthly total.
• Which model each flow uses and how often it's used.
• Percentage of requests that reuse identical or near-identical context.
• Human hours currently spent reviewing or retyping outputs for the flow.

Prioritisation matrix: which levers to pull first

I prioritise high-impact, low-integration changes first: cost-aware routing, short caches, and small batching windows. Retrieval and rule logic come next. Full redesigns and embeddings are for core flows that still dominate spend after quick wins.

Use a three-box map—Quick Wins, Worth the Work, Architectural Fixes—to assign effort.

• Quick wins (low effort, high impact): cost-aware routing, short memoisation, remove duplicate context.
• Worth the work (medium effort): batching/aggregation, replace repeated context with indexed retrieval.
• Architectural fixes (higher effort): full retrieval pipelines, background summarisation, replacing expensive models for heavy flows.

Most token waste comes from sending the same large context repeatedly, not verbose prompts.

Routing and cost-aware model selection

I route simple classification, formatting and validation tasks to cheaper models and reserve expensive models for creative or high-risk work; adding a decision step before each call saves a large share of spend with minimal code changes.

Implement a short inspector that checks request purpose and either sends it to the cheap model or escalates on failure.

• Create a short list of request types that never need the large model.
• Implement a two-step call for ambiguous items: cheap model first, expensive model on failure.
• Log model choice per request so you can calculate cost per flow.

Caching and memoisation

I cache identical requests with a short TTL so repeated queries hit the cache instead of the model; memoisation works best for idempotent calls and predictable user queries and is cheap to add behind a feature flag.

Compute a stable request signature, store the response for a short window (minutes to hours), and return cached results when signatures match.

• Cache identical prompts and small-result responses like classifications or code snippets.
• Avoid caching for time-sensitive or stateful requests.
• Track cache hit rate and tokens saved; aim for measurable hits in the first week.

Batching and aggregation

I collect similar requests for a brief coalescing window and send them as one combined model call; batching lowers per-item token overhead and reduces repeated context. Implement a buffer that groups items and unbundles the combined response back into individual results.

Start with 100–500 ms for synchronous flows and a few seconds for background jobs, then tune.

• Good for high-throughput webhook-style traffic and background jobs.
• Use batching for summary jobs and multi-item classification.
• Measure reduced token use per item after batching to confirm savings.

Retrieval vs full-context prompts

I replace repeated full-context prompts with a small retrieval step that finds only relevant passages to include; indexing the documents and returning short passages cuts context size dramatically for document-heavy tasks.

Index once, run a lightweight search for the most relevant paragraphs, and append those instead of sending whole files to the model.

• Index once and reuse; you do not need to rebuild systems to start a search layer.
• Return a short passage rather than the whole file to the model.
• Monitor answer quality; if precision drops, increase retrieved passages rather than sending the whole document.

A feature-flagged router and a short cache buy time and money faster than a rewrite.

Pre- and post-rule logic

I apply simple rules before and after model calls to filter cases that never need an LLM and to auto-accept low-risk outputs; rules remove predictable volume cheaply and protect quality when combined with sampling.

Add exact-match filters, regex checks or business-rule skips before calls and auto-accept outputs that meet confidence thresholds.

• Apply exact-match answers from a database before calling the model.
• Auto-accept model outputs that pass confidence rules to avoid human review.
• Log rule-triggered skips so you can measure what fraction of calls were avoided.

Concrete how-to patterns you can add this quarter

I recommend three patterns you can implement quickly behind feature flags: a cost router, a memo cache, and a 3-second batcher. Each is small to code, safe to test, and gives clear measurements you can roll back if needed.

Measure model mix, cache hit rate and tokens per item before and after each change.

• Cost router: inspect request type and route to cheap or expensive model. Measure: model mix percentage and cost per flow.
• Memo cache: compute request signature, return cached output on match. Measure: cache hit rate and tokens avoided.
• 3-second batcher: buffer similar requests for up to 3 seconds and send combined request. Measure: tokens per item before vs after.

Tradeoffs and guardrails: accuracy, latency and failure modes

Every cost control reduces something: accuracy, speed, or predictability. I always canary changes on a single team, run side-by-side comparisons, and keep quick rollbacks. If a cheaper model fails for some inputs, escalate just those items to the expensive model rather than reverting the whole change.

Set automated checks and human sampling to catch regressions early.

• Always start with a small traffic slice and measure accuracy on a held-out sample.
• Expect slight increases in response time for batching, but a drop in per-item cost.
• Have an automated rollback when weekly quality metrics fall below a threshold.

How these changes fail in practice and what to watch for

I see the same failure pattern: changes go live without measurement, caches become stale or too coarse, users get wrong answers, trust erodes, and the change is rolled back. The root cause is missing monitoring and no human-in-the-loop sampling for edge cases.

Instrument outputs and track support signals so you detect problems before customers complain.

• Failure mode: caching returns stale or context-inappropriate answers because request signature was too coarse.
• Failure mode: routing sends ambiguous requests to the cheap model and causes silent correctness regressions.
• Mitigation: add per-flow quality checks and sample human review for early detection.

If you cannot allocate token spend to a team within ten minutes, you cannot prioritise savings.

Monitoring and governance: what to track and how to alert

I track tokens per call, calls per flow, model mix, cost per flow, cache hit rate and human review hours. Build daily dashboards and a weekly report that assigns cost to teams so owners see their spend and can act before surprises hit finance.

Set simple alerts on sharp cost or quality deviations and attach process tags to each request.

• Daily: tokens used, calls, cache hit rate, model split.
• Weekly: cost per flow and cost per team (attach process tags to each request).
• Alerts: trigger when 7-day rolling cost for a flow exceeds 2× its baseline or when cache hit rate drops sharply.

How to allocate token costs to teams and prove savings

I add a process tag to each request and roll up tokens and price to team-level reports. Use the same pre/post time window to present monthly deltas to finance and include human-hours saved as payroll reduction using the simple team-hours calculation.

Keep the allocation simple: tag, aggregate, and report.

• Start by tagging calls with team or owner and export weekly totals.
• Create a simple chargeback spreadsheet showing monthly tokens × price per 1,000 tokens.
• When you deploy a change, compare the same week before/after to show measurable savings.

Example calculation: a 50-person SaaS and a triage flow (sample numbers)

I provide a simple example you can rerun with your inputs: take requests per month, average tokens per request, and your vendor price per 1,000 tokens to compute monthly spend. Apply an expected cache hit percentage to show tokens and dollars saved for that flow.

Substitute your real counts to reproduce the arithmetic.

• Inputs (example): 50,000 requests/month for the triage flow; average 1,200 tokens per request; price = your vendor price per 1,000 tokens.
• If you cut repeated context and cache 40% of requests, tokens saved = 50,000 × 1,200 × 40%. Multiply by price per 1,000 to get monthly dollars saved.

30/60/90 day runbook to cut token costs and prove savings

My 30/60/90 plan is practical: in 30 days measure and implement one low-effort router or cache on a single flow; in 60 days expand caches, add batching to a second flow and run a retrieval test; in 90 days roll out what worked and show the cost delta to finance.

The aim is measurable reduction and a reproducible playbook.

• Days 0–30: measure tokens per flow, pick one high-volume flow, add a cost router and short cache behind a feature flag, track baseline vs week-over-week.
• Days 31–60: add batching and a retrieval test on the second flow; start allocating costs to teams and report to finance.
• Days 61–90: full rollout of successful patterns, permanent alerts, and a quarterly review with owners to prevent regressions.

When you should call in external help

Call in external help when your highest-spend flows are core to the product or when cost controls risk customer experience and you lack technical leadership for canaries and quality testing. I run short diagnostics that map spend and deliver a 30/60/90 plan if you want outside help.

See Services for how I work and Portfolio and case studies for examples.

• Bring in advisors if you need independent measurement, a safe rollout plan, or help prioritising engineering time.
• If you want examples of this applied before, see Portfolio and case studies.
• Read about my background at About the author if you want an operator's approach.

Where businesses waste the most time

1. Re-sending full documentsHigh tokens sent repeatedly and larger bills for document workflowsAdd a short retrieval step that pulls only relevant passages and include those instead of whole filesA person still checks: Reviewer checks retrieved passages for edge cases and approves substitutions
2. Using expensive model for simple tasksPaying premium rates for classification and formattingIntroduce a small decision step that sends cheap tasks to a lower-cost model and escalates failuresA person still checks: Team member reviews escalations the cheap model could not resolve
3. Repeated identical queriesMultiple identical calls across sessions for the same answerAdd a short-lived cache keyed by a request signature so identical queries return cached responsesA person still checks: Owner spot-checks cache misses and stale results weekly

How this compares with what you already run

Operational area	The manual way	The automated way	Annual business impact
Routing/model choice	Always call the most capable model	Route simple tasks to cheaper models and escalate failures	Lower recurring model spend
Repeated context	Send full documents each time	Search and include only relevant passages	Reduce tokens per request
Caching	Recompute identical outputs every time	Return cached responses with short TTL	Stabilise monthly bills
Batching	Handle each item individually	Group similar items into periodic calls	Lower per-item overhead
Rule logic	Human or model handles routine cases	Apply rules before calling the model to skip trivial requests	Remove predictable volume from the bill
Monitoring	No per-flow tracking	Track tokens, model mix and cost per flow daily	Turn surprises into owned metrics

What the end-to-end workflow looks like

The metrics that tell you it worked

Metric	What it tells you
Tokens per call	Average tokens sent to the model per request; it falls when you reduce context or use retrieval.
Calls per flow	Number of model calls a process produces per day; it drops with caching or batching.
Model mix percentage	Share of calls sent to each model; it shows the effect of routing on cost.
Cost per flow	Tokens × price mapped to a process; the number finance cares about.
Cache hit rate	Share of requests served from cache; higher means fewer model calls.
Human review hours	Time people spend checking outputs; converts automation into payroll savings.
Requests escalated	Count of items sent from cheap model or rules to an expensive model; used to monitor quality.

When this is the wrong move

The mistakes that cost the most

• Skipping measurementTeams deploy cost changes without a baseline, then cannot prove savings or diagnose regressions; sequence: change → user complaints → rollback with no data.
• Caching too broadlyCache key is too coarse, stale answers get returned; sequence: cache added → incorrect responses → trust drops → feature removed.
• Routing everything to cheap modelAmbiguous requests fail silently because routing is undeveloped; sequence: routing added → quality drops for edge cases → customer impact.

Frequently asked questions