Business

Local LLM for business in the EU

Running a model locally is a choice about where it runs. It is not proof that you meet the GDPR or the AI Act. This page lists the questions a company still has to answer. It is not legal advice: check your answers with counsel or your data protection officer (DPO).

What “local” covers

A local model keeps the weights and the inference on hardware you control. Everything around it can still live somewhere else. Each of these has its own location, and you need to know it:

Model weights.
Prompts and the documents pasted into them.
Outputs.
Embeddings and the vector store behind a search or chat-with-your-files setup.
Logs, from the model server, the proxy and the app.
Backups of all of the above.
Telemetry sent by the runtime, the app or the operating system.
Remote support access by a vendor or an IT partner.
Tool calls: web search, plugins or APIs the model is allowed to call.

An “EU-hosted” badge answers one of these questions, not all of them.

Common ways to deploy:

A laptop or workstation per user.
An on-premises server shared by a team.
A private cloud in your own tenancy.
An EU-hosted dedicated service run by a provider.
An air-gapped machine with no network route out.
Hybrid: some requests local, some to a cloud model. This needs routing you can prove, and no silent fallback to the cloud when the local model is busy or fails.

The GDPR still applies

The GDPR is about processing personal data, wherever the processing happens. A local setup changes who processes the data and where. It does not remove these questions:

Which personal data ends up in prompts, logs and embeddings? Is any of it special-category data, such as health data?
What is the lawful basis, and what is the purpose? Is the use limited to that purpose?
Do you keep only what you need? How long do you keep logs, prompts and vector indexes, and how are they deleted?
Is the processing likely to be high risk? If so, a data protection impact assessment (DPIA) is required before you start.
Can you answer an access or deletion request that touches prompts, logs or an index?
Can data still leave the EU through support access, telemetry or a tool call?
Do employees use it? Then check internal policy, and in some countries the works council has a say.

The AI Act looks at the use case, not the model

The AI Act sorts obligations by what an AI system is used for and by your role. A company that builds a system or places it on the market is a provider. A company that uses one under its own authority is a deployer. Running the model yourself does not change which uses are allowed.

Prohibited practices: some uses are banned outright.
High-risk uses, for example in hiring, education or access to essential services, carry extra duties.
Transparency duties: people may need to be told they are dealing with an AI system or AI-generated content.
AI literacy: staff who use or run the system need a sufficient level of understanding.
Human oversight: in high-risk uses, someone must be able to check and override the output.

The obligations apply in phases. The European Commission's AI Act page says the Act entered into force on 1 August 2024, that the prohibited practices and AI literacy obligations apply from 2 February 2025, and that the obligations for general-purpose AI models apply from 2 August 2025. The same page lists later dates for transparency and high-risk rules, some of them moved by an amendment. Check the Commission timeline before you plan around a date.

The model licence

Open weights do not mean free for any business use. Each model has its own licence, and some add conditions for commercial use or for large companies. Each model page here shows a label: Commercial use yes, with conditions, or unclear. Read the licence itself before you rely on it. Examples: Llama 3.3 70B and Qwen2.5 14B.

Cost and sizing

Buying hardware only pays off with enough use. The monthly comparison on buy vs rent includes electricity. Several people at once need more memory, because each user adds KV cache. The calculator takes users at the same time; its memory and speed figures are an estimate from the formula: Llama 3.3 70B Q4 for 10 users.

A short checklist

A data map: where weights, prompts, outputs, embeddings, logs and backups live.
Which personal data and special-category data go in.
A lawful basis, and a screen for whether a DPIA is needed.
An AI Act risk screen for this use case.
Who is the provider and who is the deployer.
The model licence allows your use.
Where logs, embeddings and backups are stored, and for how long.
Who can access the system remotely, including vendors.
No cloud fallback unless it was approved.
Access control, ideally through single sign-on (SSO).
Retention periods and a way to delete.
A budget and an expected number of users and requests.

Official sources

This page is not legal advice. Take your answers to counsel or your DPO before you decide.

Find hardware

Rig Report waitlist

A written hardware plan for one model and budget. Planned price, not for sale yet.

What the Rig Report is