Guide
Running DeepSeek and Qwen locally
This site does not include the full DeepSeek-R1 mixture-of-experts model. It includes one DeepSeek distill and three Qwen2.5 sizes. Memory below is the formula at 8,192 tokens.
What is in the catalog
- Qwen2.5 14B: 48 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/Qwen/Qwen2.5-14B/raw/main/config.json. The parameter count 14.7B is still an estimate (paramsSourceUrl null).
- Qwen2.5 32B: 64 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/Qwen/Qwen2.5-32B/raw/main/config.json. The parameter count 32.8B is still an estimate (paramsSourceUrl null).
- Qwen2.5 72B: 80 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/Qwen/Qwen2.5-72B/raw/main/config.json. The parameter count 72.7B is still an estimate (paramsSourceUrl null).
- DeepSeek-R1-Distill 70B: architectureEstimated is true and architectureSourceUrl is null. A distill on a Llama 70B shape. Architecture is copied from Llama 3.3 70B and marked as an estimate: no separate config was fetched. The parameter count 70.6B is an estimate (paramsSourceUrl null).
If you wanted the original DeepSeek-R1 MoE, it is not a row here, and this page will not guess its VRAM. The distill is a 70B-shaped model with the caveat above.
Q4 memory and where it lands
- Qwen2.5 14B Q4 at 8,192 tokens is 10.6 GB in this formula (8.2 GB weights, 1.6 GB KV cache, 0.8 GB overhead). Mid-range PC with RTX 4060 Ti 16GB: a full fit. Used PC with RTX 3090 24GB: a full fit. Dual RTX 3090, 48 GB: a full fit.
- Qwen2.5 32B Q4 at 8,192 tokens is 21.3 GB in this formula (18.4 GB weights, 2.1 GB KV cache, 0.8 GB overhead). Mid-range PC with RTX 4060 Ti 16GB: offload only, not a full fit. Used PC with RTX 3090 24GB: a full fit. Dual RTX 3090, 48 GB: a full fit.
- Qwen2.5 72B Q4 at 8,192 tokens is 44.2 GB in this formula (40.7 GB weights, 2.7 GB KV cache, 0.8 GB overhead). Mid-range PC with RTX 4060 Ti 16GB: offload only, not a full fit. Used PC with RTX 3090 24GB: offload only, not a full fit. Dual RTX 3090, 48 GB: a full fit.
- DeepSeek-R1-Distill 70B Q4 at 8,192 tokens is 43.0 GB in this formula (39.5 GB weights, 2.7 GB KV cache, 0.8 GB overhead). Mid-range PC with RTX 4060 Ti 16GB: offload only, not a full fit. Used PC with RTX 3090 24GB: offload only, not a full fit. Dual RTX 3090, 48 GB: a full fit.
Qwen2.5 14B Q4 is the one that still fits the 16 GB PC. Qwen2.5 32B needs the 24 GB class. The 72B Qwen and the DeepSeek distill follow the same memory class as Llama 3.3 70B, so a single 24 GB or 32 GB card is not a full fit. Q8 and FP16 are larger; switch quant in the calculator rather than assuming Q4 is the file you have.
Tokens per second on the calculator are estimates from the methodology page. benchmarkSourceUrl stays null. This guide does not add a measured llama-bench result.
Open a model: Qwen2.5 14B, Qwen2.5 32B, Qwen2.5 72B, DeepSeek-R1-Distill 70B. Or the calculator: DeepSeek distill Q4, Qwen2.5 32B Q4.
Related: How much VRAM does a 70B model need?, Quantization explained for local LLMs, Mac vs PC for a local LLM.