Guide
How much VRAM does a 70B model need?
A 70B model at Q4 is about 40 GB of weights in this formula, before KV cache. A 24 GB card and a 32 GB card both miss that. The figures below are the formula, not a file we downloaded.
Where the shape comes from
- Llama 3.3 70B: 80 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/raw/main/config.json. The parameter count 70.6B is still an estimate (paramsSourceUrl null).
- Qwen2.5 72B: 80 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/Qwen/Qwen2.5-72B/raw/main/config.json. The parameter count 72.7B is still an estimate (paramsSourceUrl null).
- DeepSeek-R1-Distill 70B: architectureEstimated is true and architectureSourceUrl is null. A distill on a Llama 70B shape. Architecture is copied from Llama 3.3 70B and marked as an estimate: no separate config was fetched. The parameter count 70.6B is an estimate (paramsSourceUrl null).
Weights use 0.56 bytes per parameter for Q4, 0.69 for Q5, 1.07 for Q8 and 2 for FP16, plus the idea of a little container overhead. KV cache stays FP16 (2 bytes per element). Overhead is a flat 0.8 GB. None of that is a measured GGUF file size.
Llama 3.3 70B at 8,192 tokens
- Q4: 39.5 GB weights + 2.7 GB KV + 0.8 GB overhead = 43.0 GB.
- Q5: 48.7 GB weights + 2.7 GB KV + 0.8 GB overhead = 52.2 GB.
- Q8: 75.5 GB weights + 2.7 GB KV + 0.8 GB overhead = 79.0 GB.
- FP16: 141.2 GB weights + 2.7 GB KV + 0.8 GB overhead = 144.7 GB.
Qwen2.5 72B Q4 is 44.2 GB on the same context. The DeepSeek distill Q4 is 43.0 GB, using the estimated Llama-shaped architecture above. A longer context grows only the KV term. At the config max of 131,072 tokens the KV term would be much larger than at 8,192; the calculator will recompute it.
Which machines clear Q4
- GeForce RTX 3090 24GB has 22.8 GB usable after the reserve in this formula (catalog memory 24 GB, vram). Llama 3.3 70B Q4: does not fit.
- GeForce RTX 5090 32GB has 30.8 GB usable after the reserve in this formula (catalog memory 32 GB, vram). Llama 3.3 70B Q4: does not fit.
- Dual RTX 3090, 48 GB has 45.6 GB usable after the reserve in this formula (catalog memory 48 GB, vram). Llama 3.3 70B Q4: full fit.
- Mac mini M5 Pro 64GB has 54.0 GB usable after the reserve in this formula (catalog memory 64 GB, unified). Llama 3.3 70B Q4: full fit.
- Mac Studio M4 Max 128GB has 116.0 GB usable after the reserve in this formula (catalog memory 128 GB, unified). Llama 3.3 70B Q4: full fit.
A discrete GPU subtracts 1.2 GB per card before that usable figure. A Mac subtracts a system reserve instead (10 GB at 64 GB, 12 GB at 128 GB). Offload, when it exists, uses system RAM after leaving 8 GB for the OS, and the calculator will not prefer it while a full fit exists.
Tokens per second on the calculator are estimates from the methodology page. benchmarkSourceUrl stays null. This guide does not add a measured llama-bench result.
Check a machine: Llama 3.3 70B Q4, Qwen2.5 72B Q4, Llama 3.3 70B model page, the €2,000 Llama 70B landing.
Related: Best GPU for local LLMs in 2026, Mac vs PC for a local LLM, Quantization explained for local LLMs.