localLLMriglocal, on your table

Guide

Quantization explained for local LLMs

Quantization, in this project, is only a bytes-per-parameter assumption. It is close to a GGUF label and it is not a file we weighed. We also have no quality benchmark, so this page will not tell you how many points Q4 drops.

Four rates, all assumed

  • Q4: 0.56 bytes per parameter. The methodology page calls this close to GGUF Q4_K_M, including a little container overhead.
  • Q5: 0.69 bytes per parameter, close to Q5_K_M on that same page.
  • Q8: 1.07 bytes per parameter, close to Q8_0.
  • FP16: 2 bytes per parameter.

The parameter count itself is not in the Hugging Face config, so even a model with a config URL still has paramsEstimated true. KV cache does not use these rates. It stays FP16 in the formula, so a heavier quant shrinks weights and leaves the KV term alone.

The same model, four sizes

Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct/raw/main/config.json. The parameter count 8.03B is still an estimate (paramsSourceUrl null). Llama 3.3 70B: 80 layers, 8 KV heads, head dim 128, max context 131,072. Those fields come from https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/raw/main/config.json. The parameter count 70.6B is still an estimate (paramsSourceUrl null).

  • Llama 3.1 8B Q4: weights 4.5 GB, total with KV and overhead 6.4 GB.
  • Llama 3.3 70B Q4: weights 39.5 GB, total 43.0 GB.
  • Llama 3.1 8B Q5: weights 5.5 GB, total with KV and overhead 7.4 GB.
  • Llama 3.3 70B Q5: weights 48.7 GB, total 52.2 GB.
  • Llama 3.1 8B Q8: weights 8.6 GB, total with KV and overhead 10.5 GB.
  • Llama 3.3 70B Q8: weights 75.5 GB, total 79.0 GB.
  • Llama 3.1 8B FP16: weights 16.1 GB, total with KV and overhead 17.9 GB.
  • Llama 3.3 70B FP16: weights 141.2 GB, total 144.7 GB.

Budget PC with RTX 3060 12GB usable memory is 10.8 GB. Llama 3.1 8B Q4 is a full fit there; Q8 is a full fit. Dual RTX 3090, 48 GB usable memory is 45.6 GB. Llama 3.3 70B Q4 is a full fit; Q8 is offload only, not a full fit; FP16 is offload only, not a full fit. Mac Studio M5 Ultra 256GB is the catalog row whose summary says it can still hold 70B in FP16: the formula calls that fit a full fit.

What quantization is not, here

It is not a measured file size, not a perplexity score, and not a leaderboard. Q3 and Q6 are not in the formula. If you need a quality comparison, it is not on this site.

See the effect on a real pick: 70B at Q4, 70B at Q8, 8B at Q4, methodology.

Related: How much VRAM does a 70B model need?, Running DeepSeek and Qwen locally, Best GPU for local LLMs in 2026.

Open the calculator

Quantization explained for local LLMs · localLLMrig