estimate, except measured file sizes
Can I run Llama 3.1 8B locally?
8.03B parameters. Parameter count is an estimate.
- Licence:
- not recorded
- Context:
- 131,072 tokens
- Modality:
- not recorded
- Released:
- not recorded
The default local model. Q4 fits in 8–12 GB, Q8 wants closer to 12–16 GB.
· Architecture source · Model page
Memory per quant
KV is at 8,192 tokens. Overhead is 0.8 GB. Both are estimates. A measured file size is labelled. The last column is the cheapest machine that fits the total fully.
| Quant | File size | KV at 8K | +0.8 overhead | Total | Cheapest full fit |
|---|---|---|---|---|---|
| Q8 | 8.6 GB est. | 1.1 GB est. | 0.8 GB est. | 10.5 GB est. | GeForce RTX 3060 12GB, €257 (used (median of sold listings)) |
| Q4 | 4.8 GB est. | 1.1 GB est. | 0.8 GB est. | 6.7 GB est. | GeForce RTX 3060 12GB, €257 (used (median of sold listings)) |
Cheapest rig that fits
Q4, full: GeForce RTX 3060 12GB, €257 (used (median of sold listings))
Q4, slow, partly in system RAM: Nothing on our list of 29 machines fits this quant.