A UK CTO we spoke to last month was evaluating 5 agencies for a serious AI build. All five had polished AI capability slides. Three had case studies claiming enterprise AI deployment. Two showed impressive-looking chatbot demos. He could not tell which had real production AI experience and which had built ChatGPT wrappers with marketing spin.
That is the AI vendor selection conversation across UK and US enterprises in 2026. Every agency claims AI capability now. Some have production experience across multiple deployments; many have shipped one chatbot and rebranded as an AI shop. The gap between real AI capability and wrapper factory is invisible from the pitch deck. Ten specific questions reveal the reality.
This article is a candid checklist for CTOs, CEOs, and heads of AI evaluating agencies. Ten questions that separate production AI shops from wrapper factories. Red flags and green flags. What production AI actually looks like. Written from the buyer's side, though we are ourselves an AI development company.
The 10 Questions That Reveal Reality
Ten specific questions that surface real capability.
1. Show me a production AI system beyond a chatbot. Chatbot demos are cheap. Production AI beyond chatbots (agents with tool use, RAG systems, fine-tuned models, multi-modal reasoning) requires actual capability.
2. What evaluation harness do you use? Production AI requires evaluation harness with golden test sets, regression detection, and continuous quality monitoring. Real answers name specific tools (LangSmith, Weights & Biases, Promptfoo, custom harness) and describe workflow.
3. What is your model tier routing strategy? Real AI shops route by task complexity: fast tier (Haiku, GPT-4o mini) for simple work, flagship (Opus, GPT-4, Gemini flagship) for hard tasks. Cost per request drops 3-5x with quality preserved.
4. How do you handle prompt caching? Both Anthropic and OpenAI offer prompt caching with 50-90 percent discount on cached tokens. Real AI shops implement this for cacheable context (system prompts, tool definitions, RAG context).
5. What is your MLOps stack for AI-specific concerns? Model version tracking, prompt version tracking, evaluation history, cost monitoring, error rate tracking. Real answers name tools and describe workflow.
6. How do you handle hallucinations in production? Real answers discuss RAG for grounding, output validation, confidence scoring, human-in-the-loop where appropriate, evaluation harness catching hallucination regression.
7. What is your production AI cost per user or per request? Real AI shops track cost per user and cost per request. They can quote typical ranges from real deployments.
8. Which model providers do you have production experience with? Real answers include multiple providers (Claude, GPT, Gemini, sometimes Llama). Single-provider agencies are locked into one vendor's economics.
9. What happens when a model provider deprecates a model? Anthropic and OpenAI deprecate models periodically. Real AI shops have evaluation harness to test on new model, migration plan, and rollback capability.
10. Show me a RAG or fine-tuning production system you shipped. RAG and fine-tuning are the two mainstream advanced patterns beyond simple prompting. Real AI shops have shipped at least one of each in production.
The Gartner generative AI research and Stanford AI Index report reinforce these capability benchmarks.
Red Flags: AI Wrapper Factory Signals
Eight signals of wrapper factory disguised as AI shop.
Only shows chatbot demos. Chatbots are the easiest AI product.
Cannot explain evaluation harness. Golden test sets, regression detection, continuous monitoring.
Uses only one model provider. Real AI shops are provider-neutral.
No monitoring or drift detection in production. Model quality degrades over time.
No cost tracking or optimisation. Prompt caching, model tiering, batch API.
Cannot answer "what happens when GPT-4 gets deprecated". No migration playbook.
Uses hallucination as a product feature. Red flag for production systems handling real user data.
Charges by hour without production capability. T&M rates without capability to justify them.
Any 3+ together signal wrapper factory. Walk from these evaluations.
Green Flags: Real AI Shop Signals
Eight signals of production AI capability.
Ships production AI beyond chatbots. Agents with tool use, RAG systems, fine-tuned models, multi-modal reasoning.
Has evaluation harness with golden test sets. Named tools, described workflow, examples of catching regressions.
Uses multi-model tiering. Cost per request drops 3-5x. Real numbers from production.
Discusses prompt caching and cost optimisation. Understands the ecosystem, quotes real savings.
Has MLOps stack. Named tools (LangSmith, Weights & Biases, Promptfoo, or equivalent).
Handles hallucinations as first-class engineering concern. RAG grounding, output validation, confidence scoring.
Tracks production AI cost per user. Real numbers from real deployments.
Migrates between model providers routinely. Has done it, has playbook, is not locked in.
Agencies with 6+ of these are real AI shops. Serious evaluation candidates.
What Production AI Actually Looks Like
Production AI in 2026 goes beyond chatbots.
RAG systems. AI that reasons across enterprise data (Slack, GitHub, documents, databases). Vector database plus retrieval logic plus LLM generation. Real deployments handle citations, freshness, and access control.
AI agents with tool use. AI that takes multi-step actions using external tools (calendar, email, CRM, code execution, browser). Not chat responses; actual task completion.
Fine-tuned models. Custom models for specific business tasks (specific vocabulary, format enforcement, cost optimisation at scale). Requires evaluation harness and retraining discipline.
Multi-modal capability. Text + image + video + document reasoning where use case demands. Frame sampling for video, layout parsing for documents.
Evaluation and continuous monitoring. Golden test set re-run on model updates. Cost per request tracked.
Error rate monitored. Drift detected before users notice.
Agencies unable to demonstrate these capabilities in production are wrapper factories regardless of what their pitch deck claims.
What We See in the Market
Across AI development engagements at WhiteStone, three patterns recur.
The market is full of wrapper factories. Most agencies added "AI capability" to their pitch in 2024-2025. Few developed real production AI experience beyond chatbots. Structured evaluation surfaces this quickly; unstructured evaluation does not.
Real production AI experience is uncommon and increasingly valuable. Agencies with genuine production AI experience across multiple deployments command materially higher rates and get selected on capability rather than price.
Buyers who use structured evaluation pick better regardless of vertical. Buyers who apply the 10-question checklist (whether ours or their own) end up with more successful AI engagements even when they choose a different agency. Structure beats gut feel.
We built IELTSArena as production AI shipping to real users daily. It uses multi-model tiering (fast tier for grammar checks, flagship for band-descriptor evaluation), prompt caching on IELTS context, RAG grounding on band descriptors, and evaluation harness re-run on every model update. This is what production AI capability looks like. See our earlier post on MCP (Model Context Protocol) Explained for the broader capability ecosystem shaping 2026 AI development.
You can see IELTSArena at our portfolio. If you want a candid conversation about AI development for your specific need, book an AI development call with WhiteStone.
Common Failure Modes
Three failure modes we see repeatedly.
Selecting on pitch deck rather than production reference. Every agency has a beautiful AI slide. Only some have production systems to show. Ask for production references before contract signing.
Ignoring model deprecation risk. Agency locks you into one provider. Provider deprecates model 12 months into engagement. Agency scrambles without migration plan. Delivery slips.
Accepting "AI is creative" as excuse for hallucinations. Production AI handles real user data. Hallucination on factual claims is engineering failure, not creative feature. Agencies dismissing this are unprepared for production.
Frequently Asked Questions
How do I tell an AI agency from an OpenAI wrapper factory?
Ten questions reveal reality: production AI beyond chatbot, evaluation harness, model tier routing, prompt caching, MLOps stack, hallucination handling, cost per user tracking, multi-provider experience, model deprecation planning, and shipped RAG or fine-tuning systems. Agencies passing 6+ are real; agencies passing 3 or fewer are wrapper factories.
What real AI work should an agency be able to show?
Production AI beyond chatbots: agents with tool use, RAG systems reasoning across enterprise data, fine-tuned models for specific business tasks, multi-modal capability where relevant. All with evaluation harness and continuous quality monitoring. Agencies that only show chatbot demos are wrapper factories.
What questions should I ask about production AI cost?
What is your production AI cost per user or per request. Which cost optimisation techniques do you use (prompt caching, model tiering, batch API, semantic caching). What is your typical cost trajectory from prototype to production. Real AI shops answer with specific numbers from real deployments.
How do agencies handle model provider deprecation?
Real AI shops have evaluation harness to test on new model, migration plan, and rollback capability. They have done it before and have playbook. Ask "what happens when GPT-4 gets deprecated" and evaluate the answer for evidence of real migration experience versus theoretical planning.
Should we always avoid AI agencies that use only one model provider?
Not always but usually. Single-provider commitment signals capability gap (never tested others) or vendor lock-in. Real AI shops are provider-neutral because different providers have different strengths for different workloads. Single-provider agencies should be probed for why; if the answer is not compelling, walk.
The One Thing to Remember
Every agency claims AI capability in 2026. Real production AI experience is uncommon and increasingly valuable. Ten specific questions separate production AI shops from ChatGPT wrapper factories. Buyers who use structured evaluation pick well regardless of vertical or budget. Buyers who go on pitch quality often pick wrong and spend the next 6-12 months paying for a wrapper factory dressed as AI shop.
If you want a candid conversation about AI development for your specific need, browse our AI development services or come to the call.
.webp)
.webp)