Private LLM: How to Run Your Own Model On-Prem or in a VPC (2026 Guide)

A private LLM is a large language model that runs where you control the data path: on your own servers, in your own cloud account, or on a provider endpoint that is network-isolated and contractually barred from training on your data. You choose an open-weight model, a serving stack and hardware, then add security controls.

That definition covers four very different setups, and the differences matter more than the label. A bank running an open model in an air-gapped room and a startup calling Claude through AWS PrivateLink can both say they use a “private LLM.” Only one of them holds the model weights.

This guide explains the four deployment options, the open-weight models worth running in November 2026, the software and hardware you need, what it costs, and when private is not worth it. Prices were checked between October 24 and November 2, 2026. Vendor claims are labeled as such, and the cost model at the end lists every assumption.

What “private LLM” means: four deployment options

Every private LLM answers two questions: where does inference run, and who can see the prompts? Here are the four answers in use today.

1. Fully on-prem or air-gapped. You buy GPUs, put them in your data center and run open-weight models with no outside connection. Closed models can now go there too: Google made Gemini on Google Distributed Cloud air-gapped generally available in August 2025, and in October 2026 it added open-weight model support and Gemini enterprise search (preview) to the air-gapped product.

2. Your own cloud VPC with self-hosted open weights. You rent GPUs from AWS, Azure, Google Cloud or a GPU cloud such as CoreWeave and run the model in your account. The data never goes to a model vendor, but it does sit on the cloud provider’s hardware.

3. Managed private endpoints. You call a hosted model through private networking. Azure OpenAI supports virtual networks and private endpoints, and Microsoft says prompts and completions are not available to OpenAI or used to train foundation models. Amazon Bedrock works over AWS PrivateLink without an internet gateway, and AWS says model providers have no access to prompts or logs. On Google Cloud, VPC Service Controls keep inference requests inside a perimeter, and Google’s training restriction bars it from training on customer data without permission.

4. Vendor-dedicated deployments. The model maker serves you under stricter terms. OpenAI offers data residency in 10 regions, Zero Data Retention by approval, and a “Private Inference” option in preview that uses confidential computing. Anthropic offers zero data retention for qualifying API customers. Cohere’s Model Vault runs its models in a single-tenant environment.

Deployment options compared

Two details trip teams up. First, managed endpoints are private in transit, not in custody: the provider still processes your data. OpenAI keeps abuse-monitoring logs for up to 30 days by default unless you are approved for Zero Data Retention. Second, residency costs extra. OpenAI charges a 10% uplift on data residency endpoints for models released on or after March 5, 2026, per the same page.

Leading open-weight models in November 2026

Options 1 and 2 need open weights. The field has changed fast. The strongest open models now come mostly from Chinese labs, and Meta’s Llama line has been replaced by Muse. Meta’s chief AI officer said the company will generally not open its most powerful models.

The table below uses Hugging Face model records read on November 2, 2026. Memory figures are our arithmetic for the weights alone (1 byte per parameter at FP8, about 0.5 at 4-bit). Mixture-of-experts (MoE) models must keep every expert in memory even though only a few run per token.

Read the license before you pick. MIT and Apache 2.0 are the simplest cases. Custom licenses mostly bite on companies that resell model access. Meta’s Muse Model License requires a “Built with Muse Spark Open” notice and a separate agreement for model-as-a-service businesses above $20 million in revenue. Qwen’s Community License and the Kimi K3 license add similar conditions, per a license review by BenchClaw. Mistral Large 4 needs a commercial license for model-as-a-service use.

The serving stack

The inference engine turns weights into an API. Almost every option below exposes an OpenAI-compatible endpoint, so applications can switch between a private model and a public API with a URL change.

• vLLM is the default for multi-user GPU serving. It is Apache 2.0, reached version 0.32.0 on October 18, 2026, and gained a Google TPU backend on October 20. Its creators founded Inferact to sell a supported version.
• SGLang is the main alternative. Its RadixAttention reuses cached work across requests that share a prefix, which helps RAG and agents. The project says it runs on more than 400,000 GPUs.
• TensorRT-LLM is NVIDIA’s engine and posts the highest NVIDIA numbers. Its own table shows Llama 3.3 70B at 2,209 output tokens per second per GPU on H100 at FP8 and 6,920 on B200 at FP4.
• NVIDIA NIM packages models with optimized engines as containers. It is free for development, but production needs NVIDIA AI Enterprise, listed at $4,500 per GPU per year.
• Red Hat AI Inference is a hardened vLLM with llm-d (1.0 in October 2026) for KV-cache-aware routing across a Kubernetes cluster. It ships inside Red Hat AI 3 alongside OpenShift AI.
• Ollama and llama.cpp run quantized GGUF models on laptops, Macs and CPUs. They are great for development and single users. Ollama handles one request per model at a time by default, and its local API has no authentication, so never expose it to a network as-is.
• Hugging Face TGI is in maintenance mode. Hugging Face itself recommends vLLM, SGLang, llama.cpp or MLX. Do not start new projects on it.

Hardware sizing

Size for memory first, then for throughput. A rough rule: weights at the chosen precision, plus 10% to 30% for the KV cache and runtime, more for long contexts and many concurrent users. Then add a second replica if the service must stay up during maintenance.

Cloud prices are on-demand list rates checked November 1, 2026, from CoreWeave, AWS, Azure and Google Cloud, reused from our TPU vs GPU comparison. B200 server prices are a Lenovo price of $550,475 and an Exxact configuration at $390,704. Memory shortages have pushed workstation prices up sharply in 2026. The RTX PRO 6000 nearly doubled from its $8,565 launch price, and the 128GB DGX Spark rose from $3,999 at launch.

What a private LLM costs: a worked TCO example

We modeled one realistic case: serving Llama 3.3 70B at FP8 with vLLM on an 8-GPU H200 node, the smallest H200 unit most clouds rent. SemiAnalysis’s InferenceX measured 2,099 tokens per second per H200 on this model at 50 tokens per second per user, with 8,000 input and 1,000 output tokens per request. One node therefore handles about 44 billion tokens a month at full load, or about 26 billion at a more realistic 60%.

API prices come from our AI API Price Index, blended at the same 8:1 input-to-output ratio.

Assumptions (change them for your case):

• Utilization: 60% of measured capacity, 730 hours a month.
• On-prem server: $370,000 (midpoint of the posted $320K-$420K range), depreciated over 36 months; 8% a year for hardware support; $1,500 a month for rack space; power at 70% of the 10.2 kW DGX H200 maximum, PUE 1.4, $0.12 per kWh.
• Cloud: CoreWeave on demand at $50.44 per node-hour; Google Cloud 3-year commitment at $37.21 per node-hour.
• Staff: 0.5 of a $220,000-a-year platform engineer for cloud, 0.75 for on-prem.
• Software: open-source vLLM, so no license fee. NVIDIA AI Enterprise would add $3,000 a month for 8 GPUs.

The result. Against a premium API like Claude Sonnet 5.5, self-hosting breaks even at about 10 billion tokens a month on owned hardware, 12.6 billion on a 3-year cloud commitment and 15.9 billion on demand. Ten billion tokens is roughly 37,000 requests a day of this size. Against GPT-5.6 Terra the break-even is slightly lower, at 9.3 billion on-prem.

Against cheap APIs, self-hosting never breaks even in this model. Our best case costs $1.09 per million tokens, about nine times the $0.12 that hosted providers charge for the same Llama 3.3 70B. Even before staff costs, an owned node at 60% load works out to about $0.57 per million tokens. Large providers own GPUs at hyperscaler volume and batch traffic from thousands of customers; InferenceX puts their cost for this model at $0.16 per million tokens.

So the honest framing is this: a private LLM pays for itself when it replaces premium API spend at high, steady volume, or when the data simply cannot leave. It rarely wins on cost against small frontier-class APIs. Run your own numbers in our AI token counter and cost calculator.

Security checklist

Running the model yourself moves security work onto your team. The OWASP Top 10 for LLM Applications lists prompt injection, sensitive information disclosure and supply chain risk as the top three, and none of them go away because the model is private.

• Verify the weights. Download from the publisher’s official repository, pin the revision hash and scan files. Prefer safetensors over pickle formats.
• No public endpoints. Bind inference servers to private subnets. Put an authenticating gateway in front; Ollama and some engines ship without auth by default.
• Identity and quotas. Issue per-team API keys or SSO tokens, with rate limits and spend caps per key.
• Log carefully. Record who asked what and which model answered, but treat prompt logs as sensitive data with retention limits and encryption.
• Guard inputs and outputs. Filter for PII, secrets and prompt injection, especially in RAG pipelines that pull untrusted documents.
• Lock down retrieval. Enforce document-level permissions in the vector store so the model cannot surface files a user could not open.
• Limit agent actions. Give tools least privilege and require human approval for writes, payments and external messages.
• Patch the stack. Serving engines ship CVEs like any web server. Track releases for vLLM, SGLang, drivers and container images.
• Encrypt everything. Disk encryption for weights and caches, TLS inside the cluster, customer-managed keys where the cloud offers them.
• Red-team before launch. Test jailbreaks, data extraction and tool misuse, then repeat after each model upgrade.

Our enterprise AI security guide covers each control in more depth.

Compliance and data residency

Self-hosting simplifies GDPR questions about processors and international transfers because no third party processes the prompts. It does not remove your own obligations as controller: lawful basis, minimization, retention and data subject rights still apply to prompts and logs.

Under the EU AI Act, obligations for providers of general-purpose AI models have applied since August 2, 2025. Fine-tuning an open model does not usually make you a provider: the Commission’s guidelines set an indicative threshold of more than one-third of the original training compute. The Act became broadly applicable on August 2, 2026, and the Digital Omnibus moved high-risk obligations to December 2, 2027 for Annex III systems and August 2, 2028 for AI in regulated products. If your private LLM makes hiring, credit or similar decisions, the high-risk rules apply no matter where it runs. Our AI governance tools guide compares software for that evidence trail.

Vendors and platforms

If you do not want to assemble the stack yourself, these vendors sell it packaged. Facts are from their own announcements.

• Red Hat AI. RHEL AI, OpenShift AI and Red Hat AI Inference, built on vLLM and llm-d. Since April 2026 the inference product runs on any managed Kubernetes, starting with CoreWeave and Azure AKS.
• NVIDIA AI Enterprise. The software layer (NIM, NeMo, drivers, support) under most on-prem builds, priced per GPU.
• Dell AI Factory with NVIDIA. Dell reported more than 6,500 enterprise adopters on October 21, 2026, up from 5,000 a quarter earlier.
• HPE Private Cloud AI. A turnkey system co-engineered with NVIDIA. HPE added governed agent access and GPU flexibility in October 2026, and says it has more than 200 customers.
• Cohere. North 2 deploys in the cloud, on-prem or air-gapped, and Cohere supports private deployments in any major cloud VPC.
• Mistral. Ships open weights, and in September 2026 partnered with Cloudera to run its models on-prem and air-gapped.
• IBM watsonx. The software edition runs on your own OpenShift cluster, with Apache 2.0 Granite models.
• Google Distributed Cloud. The main way to get a closed frontier model (Gemini) inside your own walls.

For a scored comparison of the broader platforms, see our Enterprise AI Platform Index.

When a private LLM is NOT worth it

Skip self-hosting, or start with a managed private endpoint, when:

• Volume is low or spiky. Below a few billion tokens a month, a GPU node sits idle most of the time and APIs win easily.
• You need the best model. The top closed models still lead many tasks, and the best open models need 400GB to 1TB of GPU memory.
• Contract terms already solve the problem. If your legal team accepts Bedrock, Azure or Vertex with private networking and no-training terms, you get most of the privacy for a fraction of the work.
• You lack an owner. Someone must patch, monitor, upgrade models and answer pages at 2 a.m. Without that person, private becomes fragile.
• Cheap APIs fit the task. At $0.14 per million tokens for frontier-class small models, the savings case is gone unless data rules force your hand.

Decision guide

1. Can the data leave your network at all? If no, go on-prem or air-gapped (open weights, or Gemini on GDC for a closed model).
2. Will a cloud provider under contract do? If yes, and volume is modest, use a managed private endpoint first.
3. Do you process more than ~10 billion tokens a month at premium-model prices? If yes, price out self-hosting in your VPC or on-prem.
4. Is an open model good enough? Test the top two or three candidates on your own tasks before buying hardware.
5. Do you have platform staff? If not, buy a packaged platform (Red Hat, Dell, HPE, Cohere) or stay managed.

Step-by-step 30-60-90 day rollout

Days 1-30: prove it. – Pick one use case with clear data limits, such as internal document Q&A. – Benchmark two or three open models against your current API on 200-500 real prompts. – Stand up vLLM or SGLang on rented GPUs in your VPC, behind a gateway with SSO. – Write the data classification rules: what can go to the private model, what can go to public APIs.

Days 31-60: harden it. – Add logging, quotas, PII filters and prompt-injection tests. – Run a load test to find real throughput and cost per million tokens on your traffic. – Complete the security review and a DPIA where GDPR applies. – Decide on buy vs rent using your measured numbers, not ours.

Days 61-90: scale it. – Move a second and third use case onto the same platform. – Add a second replica and failover; set an upgrade schedule for models and engines. – Publish an internal model catalog with approved uses. – Review the business case quarterly; API prices fall fast, so the break-even point moves.

FAQ

What is a private LLM?

A private LLM is a language model deployed so that your prompts and outputs stay under your control. That can mean running open weights on your own servers or cloud account, or calling a hosted model through private networking with contracts that forbid training on your data.

Is a private LLM the same as an on-premise LLM?

No. An on-premise LLM is one type of private LLM, running on hardware in your own facility. Private LLMs also include self-hosted models in your cloud VPC and managed endpoints such as Azure OpenAI, Bedrock or Vertex with private networking.

How much does it cost to run a private LLM?

A small model on a workstation costs a few thousand dollars. A 70B-class model on an 8-GPU H200 node costs roughly $29,000 to $46,000 a month in our model, including staff. That beats premium APIs above about 10 billion tokens a month, but not cheap APIs.

What hardware do I need for a self-hosted LLM?

Match GPU memory to model size. A 27B-31B model fits one 48-96GB card at 8-bit; a 70B model needs about 71GB at FP8 plus cache; models of 400B and up need a multi-GPU server or a 512GB Mac Studio at 4-bit.

Which open-source model is best for a private LLM for enterprise?

There is no single winner. For permissive licenses, start with Qwen3.8-27B, Gemma 4 or gpt-oss-120b on one GPU, or Qwen3.5-397B and DeepSeek V4.1 Flash on a full server. Test candidates on your own tasks.

Does a private LLM make me compliant with GDPR and the EU AI Act?

Not by itself. Self-hosting removes third-party processors from the prompt path, which helps with GDPR transfers. You still owe controller duties, and EU AI Act high-risk rules depend on the use case, not where the model runs.