On-premise LLMs: running language models on your own infrastructure

Why on-premise is now realistic

Two things have changed. First, open-weight models — model families published with their weights, which can be downloaded and run locally — have become good enough for the everyday tasks that matter in a company: searching and summarising internal documents, drafting, comparing versions, translating. Second, serving software has matured: inference servers such as vLLM, llama.cpp or Ollama run these models efficiently on a single machine and expose a standard API to applications.

On-premise does not mean frontier-scale. It means choosing a model that is good enough for a defined set of tasks, sized for your hardware and your number of users, and keeping full control of where data goes.

What "on-premise" should guarantee

A private deployment is only as private as its weakest link. Check that:

• Documents, prompts and answers never leave your infrastructure. No calls to external APIs, no telemetry to model vendors.
• The model is not trained on your data. It is used as published; any adaptation stays in house.
• Access follows your existing permissions. Users authenticate with the company directory and only see what they are already entitled to.
• Everything is logged. Who asked what, when, and which documents were used — stored on your systems, under your retention policy.

These are the points a data protection review will ask about. Writing them down once, with the architecture and data flows, is what turns a blocked tool into an approved one — see our notes on nLPD and GDPR for AI projects.

Sizing the hardware

Memory is the first constraint. A model needs enough memory for its weights plus a working area (the context cache) that grows with the length of documents and the number of simultaneous users. Two levers matter:

• Model size. Smaller models answer faster and fit on modest hardware; larger models handle harder reasoning and longer documents.
• Quantisation. Storing weights at lower precision reduces memory substantially, with a quality cost that is often acceptable for document tasks — and should be measured, not assumed.

In practice there are two families of deployment:

The honest way to size is to run a pilot on one machine with real documents and real users, measure response times under realistic load, and only then decide what production needs.

Connecting the model to your documents

A model on its own knows nothing about your company. The useful pattern is retrieval-augmented generation: documents are indexed locally, the relevant passages are retrieved for each question, and the model answers from them with citations. The index, the embeddings and the vector store all run inside the same perimeter as the model.

Integration usually covers:

• document repositories and file shares, read-only;
• the company directory for authentication and permissions;
• an internal web interface, and an API for other applications.

Air-gapped or controlled access

Two modes cover most needs:

• Air-gapped: no inbound or outbound connection. Updates — new model versions, security patches — arrive on controlled media in agreed maintenance windows. Suited to the most sensitive environments.
• Controlled access: the system sits on the internal network, administrative access goes through authorised, logged channels, and nothing talks to external AI services.

Operating it

A private AI system is software in production, and needs the same care:

• Monitoring of response times, errors and usage.
• Updates to models and serving software, tested against a fixed set of evaluation questions before release.
• Capacity planning as usage grows.
• Support with defined response times.

Is it worth it?

On-premise AI makes sense when confidentiality is the constraint that blocks adoption — healthcare, financial institutions, law firms, public administration, and any company with sensitive intellectual property. When data may be processed in the cloud under the right conditions, a sovereign cloud deployment with a Swiss or European provider can be a simpler alternative.

Where to start

Identify one document collection, one group of users and the confidentiality rules that apply. A pilot on a single machine, tested by your people, answers most questions before any significant hardware purchase. That is how Privatim, our private AI offering, starts.