A CTO we spoke to last week had a working customer support automation on Claude Sonnet. Six months earlier, monthly bill was $4k. Last month it was $32k. Product usage had grown, feature scope had expanded, and nobody had touched the cost architecture since the initial ship.
That is the LLM cost conversation across UK and US enterprises in 2026. The AI feature works. Usage grew. The bill grew faster. The CTO is now asked why the AI budget is 4x last quarter's forecast and whether it can be brought down without breaking the product.
This article is a candid playbook for CTOs, FinOps leads, and AI Leads reducing production LLM costs in 2026. Eight cost levers that actually work. Real savings numbers per lever. The order to apply them so you compound savings rather than fight yourself. And the failure modes that turn a cost programme into a quality regression.
The 8 LLM Cost Levers That Actually Work in 2026
Eight levers, ordered by typical cost impact.
1. Tiered model routing. Route simple tasks to fast/cheap tier (Haiku, GPT nano/mini). Route standard tasks to mid tier (Sonnet, GPT mid). Route hardest tasks to flagship (Opus, GPT frontier). Most production systems can move 60-80 percent of volume off flagship without meaningful quality loss.
2. Prompt caching. Cache the repeated portion of your prompts. Both Anthropic Claude and OpenAI offer this. Cached input tokens cost roughly 10 percent of standard input tokens. For workloads with large repeated context (system prompts, RAG chunks, agent tool descriptions), this delivers 30-70 percent cost reduction.
3. Response caching for common queries. Many workloads have long-tail query distribution. Cache responses for common queries with high semantic similarity. Cache hits cost near-zero (retrieval only, no LLM call). Typical cache hit rate 20-50 percent for customer support and knowledge base workloads.
4. Batch API for async workloads. OpenAI batch API offers 50 percent discount for non-real-time workloads with up to 24-hour processing window. Anthropic offers similar batch pricing. For overnight processing, bulk classification, and async workflows, batch API cuts cost in half without engineering complexity.
5. Structured output over free-form. Enforce JSON schema on responses. Cuts output tokens 20-40 percent by eliminating narrative padding around the actual data. Also reduces post-processing effort and validation errors.
6. Token limits per call. Set max output tokens on every LLM call. Prevents runaway generation where model produces 4k tokens when 400 would suffice. Prevents 3-10x cost spikes on edge cases.
7. RAG chunk size and retrieval optimisation. Smaller chunks with tighter retrieval reduce context token count per query. Retrieve 5 relevant chunks of 300 tokens each rather than 20 chunks of 800 tokens each. 20-40 percent context token reduction typical.
8. Fine-tuned smaller models for very high volume. Only justified above 100M tokens per month for a specific task with repeatable pattern. Fine-tuned smaller model at 50-80 percent cost reduction versus prompted flagship. See our earlier post on LLM fine-tuning vs prompting vs RAG for the decision framework.
Real Savings Numbers Per Lever
Typical impact ranges from production deployments.
Tiered model routing. 40-80 percent cost reduction on total workload. Highest impact single lever. Requires task classification logic and quality evaluation per tier.
Prompt caching. 30-70 percent cost reduction on workloads with repeated context. Highest ROI relative to engineering effort (often days to implement).
Response caching. Near-100 percent savings on cache hits. Typical cache hit rate 20-50 percent means overall savings of 15-40 percent on cached-eligible workload.
Batch API. 50 percent discount on async workloads. If 40 percent of your volume is batch-eligible, overall savings of 20 percent with minimal engineering effort.
Structured output. 20-40 percent output token reduction. Compounds with other levers.
Token limits. Prevents 3-10x cost spikes on edge cases. Not a savings per se, but a runaway prevention mechanism.
RAG chunk optimisation. 20-40 percent context token reduction. Requires retrieval quality evaluation to ensure smaller chunks do not hurt answer quality.
Fine-tuned smaller models. 50-80 percent at scale on the specific task. Requires 100M+ tokens/month to justify the £30k-£150k fine-tuning setup and ongoing retraining.
Combined applied in disciplined order, teams typically reduce monthly LLM bills 50-75 percent from unoptimised baseline. The teams that combine 5-6 levers see the biggest cumulative wins.
The Order to Apply Them
Wrong order costs money and creates thrash. Right order compounds savings.
Step 1: Set token limits on every LLM call. Immediate. Prevents ongoing runaway costs. Takes hours, not days.
Step 2: Enable prompt caching. Both vendors offer this. Enable on any workload with repeated context (system prompts, RAG, agent tool descriptions). Days of engineering effort. 30-70 percent savings on eligible workloads.
Step 3: Implement tiered model routing. Classify tasks by complexity. Route to appropriate model tier. Build quality evaluation per tier. Weeks of engineering effort. 40-80 percent structural savings.
Step 4: Add response caching for common queries. Semantic similarity matching (via embeddings) for query dedup. Cache hit reduces cost to near-zero. Weeks of engineering effort.
Step 5: Move async workloads to batch API. Identify workloads that tolerate 24-hour latency. Migrate to batch endpoints. Days of engineering effort. 50 percent savings on migrated workload.
Step 6: Refine to structured outputs. Enforce JSON schema on responses. Eliminate narrative padding. Days of engineering effort.
Step 7: Optimise RAG chunking. Smaller chunks, tighter retrieval, better relevance scoring. Weeks of engineering effort with quality evaluation.
Step 8: Consider fine-tuning. Only if steps 1-7 combined have not delivered required cost profile AND you have 100M+ tokens/month for the specific task. Months of engineering effort plus ongoing retraining.
Skip steps at your peril. Teams that jump to fine-tuning (step 8) without first exhausting steps 1-7 typically overspend on the fine-tuning programme and could have saved more with cheaper interventions.
What We Learned Deploying AI in IELTSArena
IELTSArena is our AI IELTS preparation platform. We deploy AI in production for writing feedback and speaking evaluation. Two cost lessons transfer directly.
Tiered routing saved us more than any other single decision. We route simple grammar checks to fast tier (Haiku, GPT mini equivalent). Standard essay feedback runs on mid tier (Sonnet). Complex band-descriptor evaluation with edge-case reasoning routes to flagship (Opus). Overall inference cost is roughly one-third what running everything through flagship would be. Quality is preserved on the workloads where it matters.
Prompt caching turned RAG from expensive to affordable. Our RAG layer retrieves IELTS band descriptors, examiner guidance, and sample essays for context. Before caching, every essay evaluation resent the full context (10-30k tokens). After enabling prompt caching, cached context costs 10 percent of standard rate. This one change reduced our RAG-workload cost by roughly 60 percent overnight.
You can see IELTSArena at our portfolio. If you want a candid conversation about your LLM cost optimisation, book a cost audit with WhiteStone.
Common Failure Modes
Three failure modes we see repeatedly.
Optimising the wrong lever first. Teams spend weeks fine-tuning a small model when enabling prompt caching would have delivered 50 percent savings in two hours. Order matters.
Routing to smaller models without evaluation harness. Cost drops. Quality drops silently. Users notice. Trust suffers. Every routing change needs quality evaluation on golden test set.
Ignoring token limits until a runaway happens. Model generates 8k tokens when 400 was expected. Bill spikes for one edge case. Set token limits before deployment, not after the incident.
Frequently Asked Questions
What is the single biggest LLM cost lever in 2026?
Tiered model routing. Route simple tasks to fast/cheap tier, standard tasks to mid, hardest to flagship. Most production systems can move 60-80 percent of volume off flagship without meaningful quality loss. Typical savings 40-80 percent of total workload cost. Requires task classification logic and quality evaluation per tier.
How much does prompt caching save?
Both Anthropic and OpenAI charge roughly 10 percent of standard input token cost for cached tokens. For workloads with large repeated context (system prompts, RAG chunks, agent tool descriptions), overall savings 30-70 percent. Highest ROI per engineering hour of any lever.
How do you route to smaller models without hurting quality?
Build a task classifier (rule-based or small ML model) that categorises incoming requests by complexity. Route to appropriate model tier. Maintain an evaluation harness with golden test set per tier to catch quality regressions. Start with 10 percent traffic on cheaper tier, evaluate, expand if quality holds.
Can response caching hurt user experience?
Only if cache invalidation is wrong. Response cache should invalidate when underlying data changes. For genuinely static queries (FAQ answers, definitions, standard information), cache aggressively. For queries that depend on user context or current data, cache carefully with short TTLs or dynamic invalidation.
When is fine-tuning worth the cost reduction?
Above 100M tokens per month for a specific task with repeatable pattern. Fine-tuning setup £30k to £150k, plus vendor training fees, plus ongoing retraining. At scale, fine-tuned smaller model can cost 50-80 percent less than prompted flagship. Below 100M tokens per month, prompt caching plus tiered routing usually delivers similar savings without fine-tuning cost.
The One Thing to Remember
LLM cost optimisation in 2026 is a discipline problem, not a technology problem. Eight levers, applied in the right order, typically reduce production bills 50-75 percent from unoptimised baseline. Teams that skip the cheap early levers (token limits, prompt caching, tiered routing) and jump straight to fine-tuning spend more and save less. Set the levers in order, evaluate at each step, and only escalate to the next lever when the current one is fully applied.
If you want a candid conversation about your specific optimisation, browse our AI development services or come to the audit.
.webp)
.webp)