How to run an open model on a rented GPU

A rented graphics card runs the same open models as a PC at home, and you pay by the hour. Two things matter most: a GPU with enough memory for the model, and switching it off when you are done.

1. Pick the model, then the GPU by memory

The whole model has to fit in the GPU's memory, with room for the conversation. Every figure here is for a conversation of about 6,000 words (8,192 tokens) at standard quality (Q4), and the memory is our estimate. The price is the lowest hourly price per GPU on our GPU rental price list, before VAT. Part time is 4 hours a day on 22 working days; 24/7 is 720 hours.

Check your own model first: Llama 3.1 8B Q4, gpt-oss-120b Q4, Llama 3.3 70B Q4. The cheapest card that holds a model is not always the smallest: a marketplace can price an older 48 GB card below a 24 GB one. More people at once, or longer conversations, need more memory than this.

2. Pick a provider

• Cheapest: a marketplace such as Vast.ai, or Runpod's Community Cloud. The machines belong to other companies or people, which is fine for testing but not for private data.
• Data that must stay in the EU: a company based in the EU, such as Hetzner, Scaleway, OVHcloud, Nebius or Verda. Where each provider is based.
• Billing: per second at Runpod, Vast.ai, DigitalOcean and Nebius; per minute at Lambda and Scaleway; in 10-minute blocks at Verda; by the month or hour plus a setup fee at Hetzner.

3. Start a machine with a ready template

Most providers have a ready image with the drivers installed, so you only pick it and the GPU. Two common choices:

• vLLM serves the model to programs through the same kind of API as OpenAI, and handles many requests at once. Runpod, Vast.ai and DigitalOcean offer it ready to start.
• Ollama with Open WebUI gives you a chat page in your browser and is the easiest start. Vast.ai has an Open WebUI (Ollama) template; on Runpod, Ollama installs on the PyTorch template.

Give the machine enough disk for the model download: about 42.5 GB for Llama 3.3 70B at Q4 by our estimate, plus room for the software.

4. Load the model

With Ollama, in the machine's terminal. Skip the first line on a template that already has Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen2.5:32b

Inside a container without a service manager, start the server first with ollama serve in a second terminal. The first run downloads the model, then opens a chat in the terminal.

With vLLM, start an API server. A gated model such as Llama also needs your Hugging Face token:

export HF_TOKEN=your-hugging-face-token
export VLLM_API_KEY=pick-a-long-random-key
vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ --host 127.0.0.1 --port 8000 --max-model-len 8192 --api-key "$VLLM_API_KEY"

--host 127.0.0.1 keeps the server on the machine itself, and --api-key makes every request carry the key. Ollama listens on port 11434 of the machine itself by default.

5. Connect from your own computer

The safest way is an SSH tunnel: it carries the port to your own computer, so nothing is open to the internet. Your provider shows the address and SSH port of the machine:

ssh -L 8000:localhost:8000 -p <ssh-port> root@<machine-address>

Then, on your own computer, ask the model a question through the OpenAI-style API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model": "Qwen/Qwen2.5-32B-Instruct-AWQ", "messages": [{"role": "user", "content": "Hello"}]}'

Any app that speaks the OpenAI API works with the address http://localhost:8000/v1 for vLLM, or http://localhost:11434/v1 for Ollama with -L 11434:localhost:11434 in the tunnel. Open WebUI on your own computer can use the same address.

6. Keep it private

• Ollama has no password of its own. Never open port 11434 to the internet; reach it through the tunnel.
• Some providers give each open port a public web address. Anyone with that address can then use your model, unless a key or a login sits in front.
• Personal data: prefer a provider based in the EU and sign its data processing agreement. What the GDPR still asks of you.

7. Stop paying

The meter runs while the machine is on, also when nobody uses it. Stop it when you are done. Some providers still charge for the disk of a stopped machine, so delete it to end every charge once you have copied what you need. A low prepaid balance or a spending limit stops a forgotten machine from running up a bill.

For light use, paying per token at a hosted service is often cheaper than an idle GPU. Rent a GPU or pay per token compares the two, and buy vs rent adds owning the machine.

Speeds on the calculator are our estimates; How we calculate explains them. We have not timed these models on a real machine, and this guide adds no measured result.

Related: How much VRAM does a 70B model need?, Quantization explained for local LLMs, Local LLM for business in the EU.