A UK bank CIO we spoke to last month had a genuine on-prem requirement. FCA-regulated retail bank. Customer PII in every prompt. Compliance team confirmed no cloud LLM API acceptable. His board wanted a Copilot-equivalent for internal knowledge worker productivity. He needed on-prem deployment or nothing.
That is the on-prem LLM conversation for genuinely constrained enterprises in 2026. Real regulatory obligations. Real air-gap requirements. Real reasons cloud APIs are not an option. The vendor pitches suggesting on-prem to less-constrained enterprises miss that most enterprises with "we want on-prem" preferences have no actual regulatory blocker to cloud.
This article is a candid guide for CIOs, CTOs, and Heads of AI Strategy at genuinely constrained enterprises. When on-prem is actually required. What is NOT a good reason. The 2026 hardware and software stack. Real costs. Air-gapped vs private cloud vs on-prem trade-offs. And how to think about the deployment decision.
When On-Premise Is Actually Required
Five situations where on-prem is genuinely necessary.
Strict data residency prohibiting data leaving building or country. Some regulatory environments require data not merely to stay in-country but not to leave physical infrastructure the organisation controls. Government classified work, defence contracting, some intelligence work.
Air-gapped environment. Some deployments cannot have internet connectivity at all. Critical infrastructure, defence, some healthcare imaging environments. Rules out cloud API and even private cloud.
Regulatory prohibition on third-party AI processing. Specific financial regulation, healthcare PHI restrictions in some jurisdictions, or specific government requirements prohibit processing certain data through third-party AI systems regardless of contract or SLA.
Contractual obligations to specific customers. Defence primes, intelligence contractors, and some healthcare systems have contractual obligations to keep processing within specific infrastructure. Cloud LLM APIs violate these contracts.
Latency requirements below cloud round-trip. Very rare in practice. Some real-time systems (autonomous vehicles, industrial control) genuinely need sub-100ms latency that cloud round-trip cannot meet. Almost never applies to typical enterprise workloads.
The Gartner generative AI research publishes ongoing enterprise AI deployment analysis; the pattern is consistent that on-prem is genuinely required for a specific minority of enterprise workloads, and cloud API or private cloud handles the majority.
What Is NOT a Good Reason for On-Prem
Four common motivations that do not survive TCO analysis.
"We want to save money." On-prem is usually more expensive than API at moderate scale. Break-even lands above 50-100 million tokens per month with sustained utilisation. Below that, API wins on TCO after including MLOps engineering cost.
"We want more control." Private cloud (dedicated cloud instance with residency guarantees, encryption, network isolation) provides control without full hardware ownership. Most "control" concerns are addressable in private cloud.
"We do not trust cloud." Established cloud providers (AWS, Azure, GCP) have SOC 2, ISO 27001, HIPAA, PCI DSS, FedRAMP certifications with security investment exceeding what most enterprises can afford on-prem. Trust concerns are usually stronger for on-prem than cloud when honestly assessed.
Political preference vs actual requirement. Executive team wants on-prem because it "feels safer." Technical team implements at 3-5x TCO of cloud alternative. Board discovers cost. Deployment reversed. Six months of engineering wasted.
Genuine on-prem requirements can name the specific regulation, contract, or air-gap constraint. Preference-driven on-prem cannot. Test yours before committing.
The 2026 On-Premise Stack (Hardware + Software)
Real 2026 on-prem architecture.
Hardware. NVIDIA H100 (80GB HBM3) remains mainstream in 2026. H200 (141GB HBM3e) for larger context. Blackwell B200 for newest deployments. DGX systems for turnkey deployment; custom builds for cost optimisation. See NVIDIA AI developer resources for current specifications and architecture guidance.
Serving framework. vLLM is the most popular open source serving framework in 2026 for its throughput and PagedAttention memory management. TGI (Hugging Face Text Generation Inference) is second. TensorRT-LLM for maximum NVIDIA-optimised performance.
Models. Meta Llama family for general use. Mistral or Mixtral for European deployments with strong performance-per-parameter. Alibaba Qwen for multilingual (particularly Asian languages). Cohere Command R for RAG-optimised workloads. Google Gemma as open-weights complement. Selection depends on workload; see our earlier post on open source LLMs for enterprise for detailed model comparison.
Orchestration. Kubernetes with NVIDIA GPU operator for most deployments. Bare-metal for maximum performance at cost of operational complexity. Ray for distributed inference at scale.
Monitoring and MLOps. Prometheus and Grafana for infrastructure metrics. Weights & Biases or MLflow for experiment tracking. Evaluation harness for quality monitoring. Golden test set re-run weekly to catch drift.
Air-Gapped vs Private Cloud vs On-Prem
Three deployment patterns with distinct trade-offs.
Full on-prem. Hardware in your datacentre. No cloud dependency. Complete data control. Highest capex; requires MLOps team; hardest to scale up or down. Best for high sustained volume with no cloud tolerance.
Private cloud. Dedicated cloud instance (AWS Outposts, Azure Local, GCP Distributed Cloud, or dedicated VPC) with data residency guarantees and network isolation. Data stays in your region or specific infrastructure. Lower capex than on-prem; still allows scaling. Best for enterprises needing residency without full hardware ownership.
Air-gapped. On-prem with no internet connectivity. Maximum security. Highest operational overhead (model updates require physical media transfer, no automatic security patches, no cloud services). Best for defence, intelligence, and specific critical infrastructure workloads.
Most "on-prem" requirements are actually private cloud requirements. Genuine air-gap is a small minority.
Real 2026 Cost: Hardware, Software, MLOps
Small model on-prem (7B-14B parameters).
Hardware capex: £60k to £150k (1-2 GPUs plus supporting infrastructure)
Annual opex: £30k to £60k (power, cooling, maintenance)
MLOps engineering: £120k to £250k per engineer (1 engineer typical)
Total year 1: £210k to £460k
Total 3-year TCO: £540k to £1.03m
Mid model on-prem (30B-70B parameters).
Hardware capex: £200k to £500k (multi-GPU or DGX system)
Annual opex: £80k to £150k
MLOps engineering: 1-2 engineers, £120k to £500k per year
Total year 1: £400k to £1.15m
Total 3-year TCO: £640k to £2.55m
Large MoE model on-prem (100B+ parameters).
Hardware capex: £700k to £2m (multi-GPU cluster, dedicated datacentre space)
Annual opex: £200k to £400k
MLOps engineering: 2-3 engineers, £240k to £750k per year
Total year 1: £1.14m to £3.15m
Total 3-year TCO: £2.02m to £6.5m
Compare to cloud API at equivalent workload: typically £30k to £300k annually. On-prem wins only above genuine sustained high volume (typically 100M+ tokens per month) or where regulatory constraint eliminates the API option.
What We Learned About Deployment Choices
We deploy AI in production for IELTSArena and evaluate deployment options for client engagements. Two lessons transfer.
We did NOT go on-prem for IELTSArena despite handling personal data. IELTSArena handles student essays and personal data. We use cloud LLM API (with appropriate DPA and residency controls) because there is no regulatory obligation to on-prem, MLOps cost would be disproportionate to our scale, and cloud provider security exceeds what we could achieve on-prem. The right answer is not always the most restrictive one.
When clients have genuine on-prem requirements, we design for the constraint. Some client engagements genuinely require on-prem or air-gapped deployment. When they do, we design the architecture appropriately. When they do not, we recommend cloud API or private cloud and save the client 3-5x TCO.
You can see IELTSArena at our portfolio. If you want a candid conversation about your deployment options, book an on-prem LLM call with WhiteStone.
Common Failure Modes
Three failure modes we see repeatedly.
Choosing on-prem for "control" without regulatory requirement. Preference-driven on-prem at 3-5x cloud TCO. Board discovers cost. Deployment reversed. Wasted engineering.
Underestimating MLOps engineering cost. GPU hardware quote looks affordable. Add £250k annually for MLOps engineering and the maths shifts materially. Budget realistically.
Skipping evaluation harness on-prem. Model updates in on-prem environment require deliberate deployment. Without evaluation harness, quality regressions go undetected until users complain.
Frequently Asked Questions
When must an LLM run on-prem in 2026?
Strict data residency (data cannot leave your infrastructure), air-gapped environments (no internet), regulatory prohibition on third-party AI (specific financial regulation, healthcare PHI, government classified), or contractual obligations to specific customers. Everything else can typically run cloud API or private cloud.
What is the cost of running Llama on-prem?
For a Llama 70B deployment: £200k to £500k hardware capex, £80k to £150k annual opex, £120k to £500k annual MLOps engineering. Total 3-year TCO: £640k to £2.55m depending on scale and redundancy requirements.
How does air-gapped deployment work?
On-prem hardware with no internet connectivity. Model updates via physical media transfer. No automatic security patches or cloud services. Highest security posture; highest operational overhead. Required for defence, intelligence, and specific critical infrastructure workloads.
What is the cheapest on-prem LLM stack?
Single NVIDIA L40S or A100 (£15k-£30k GPU), Ubuntu server, vLLM serving framework, open source Llama or Mistral model. Small model (7B-14B) deployment. Total hardware cost £30k-£80k for basic setup; add MLOps engineering annually.
When is private cloud better than on-prem?
Almost always when data residency is the only concern. Private cloud (AWS Outposts, Azure Local, dedicated VPC) provides residency guarantees without capex and MLOps overhead. On-prem is required only for air-gap, specific regulation, or contractual obligation to keep processing on your hardware.
The One Thing to Remember
On-prem LLM deployment is genuinely necessary for a specific minority of workloads: air-gap requirements, strict data residency prohibiting private cloud, or contractual obligations. Most enterprise "on-prem" preferences do not survive TCO analysis. Private cloud provides control and residency at lower TCO. Cloud API wins for the majority of workloads. If you have a genuine regulatory blocker, on-prem is the right answer; if you have a preference, private cloud usually is.
If you want a candid conversation about your deployment options, browse our AI development services or come to the call.
.webp)
.webp)