Should we use a public LLM or a private one?
Your board wants AI, compliance wants to know where the data goes, and every vendor is pushing a single-model answer that happens to be their model.
For most companies the answer is both, routed by data sensitivity rather than chosen once. Public frontier models are stronger at general reasoning and cheaper to start with; private or self-hosted models keep regulated and confidential data inside your boundary. The architecture that survives contract review is a routing layer that sends each request to the appropriate model, so the decision is made per request rather than per company — and so switching providers later is a configuration change, not a rebuild.
The classification that drives the routing is simpler than it sounds: sort your data into what could appear in a press release, what is merely internal, and what is regulated or contractually restricted. The first tier can go to a public model today. The third should not, regardless of a vendor's current terms. Most of the value sits in the first two tiers, which is why waiting for a private deployment before starting usually costs more in delay than it saves in risk.
| Criterion | Public frontier model | Private or self-hosted |
|---|---|---|
| Reasoning quality | Strongest available today | Behind the frontier, closing |
| Cost to start | Low — usage-based, no infrastructure | High — infrastructure and operations from day one |
| Data boundary | Leaves your environment | Stays inside your boundary |
| Right for | Public and internal-only data | Regulated or contractually restricted data |
| Compliance review | Depends on vendor terms, which change | Defensible by construction |
| Switching cost | Low behind a routing layer | High — infrastructure is committed |
For most companies the answer is both, routed per request by data sensitivity. Sort data into what could appear in a press release, what is merely internal, and what is regulated — the first tier can go to a public model today.
What actually forces a private deployment
- The question is usually framed wrong. It is not public versus private; it is which classes of data may leave your boundary, and under what contractual terms.
- Single-vendor decisions become architecture. Build directly against one provider's SDK and switching later means rewriting, which is exactly the lock-in the pricing assumes.
- Free tiers usually train on your data. Paid tiers of the major providers contractually do not. That distinction is frequently missed in a pilot and inherited into production.
How to choose between public and private LLMs
-
01
Classify your data first: public, internal, confidential, regulated. This is a one-afternoon exercise that determines the whole architecture.
-
02
Put a routing layer between your application and any model. Even a thin one preserves your ability to switch.
-
03
Confirm the contractual terms of whatever tier you use. On free tiers, submitted content typically improves the provider's products.
-
04
Baseline a real workflow before the pilot. Without a before number, 'AI helped' is unfalsifiable.
-
05
Keep a human review step wherever an output reaches a customer or a financial record.
Routing is straightforward to design and awkward to retrofit. Teams that build directly against one provider's SDK discover the cost when pricing changes, a model is retired, or compliance asks a question they cannot answer. We build the abstraction first for exactly this reason — our own chatbot runs provider-agnostic, and switching between Gemini, Anthropic and an OpenAI-compatible endpoint is an environment variable, not a project.
More on public and private LLMs
For many production tasks, yes — classification, extraction, summarisation and routing are well served by smaller open-weight models at a fraction of frontier cost. Frontier models earn their price on open-ended reasoning.
Marginally at the start, substantially less later. The abstraction is perhaps a week of work up front and saves a rewrite when — not if — you change providers.
No. That is precisely what the routing layer prevents. Regulated and confidential classes can be pinned to a private or self-hosted model while everything else uses whichever frontier model is best value.
More on AI Transformation
Still not sure this is your problem?
A 20-minute fit check. We will tell you if it is something you can fix without us — that happens often enough that we lead with it.
Book a Strategy Session →Browse every answer.
The full Solution Desk — specific questions, straight answers, no gate.
Open the Solution Desk →