estimate, except measured file sizes
Can I run Mistral 7B v0.3 locally?
7.25B parameters. Parameter count is an estimate.
- Licence:
- not recorded
- Context:
- 32,768 tokens
- Modality:
- not recorded
- Released:
- not recorded
Architecture is an estimate of the familiar 7B shape. No config was fetched for this dataset.
Memory per quant
KV is at 8,192 tokens. Overhead is 0.8 GB. Both are estimates. A measured file size is labelled. The last column is the cheapest machine that fits the total fully.
| Quant | File size | KV at 8K | +0.8 overhead | Total | Cheapest full fit |
|---|---|---|---|---|---|
| Q8 | 7.8 GB est. | 1.1 GB est. | 0.8 GB est. | 9.6 GB est. | GeForce RTX 3060 12GB, €257 (used (median of sold listings)) |
| Q4 | 4.4 GB est. | 1.1 GB est. | 0.8 GB est. | 6.2 GB est. | GeForce RTX 3060 12GB, €257 (used (median of sold listings)) |
Cheapest rig that fits
Q4, full: GeForce RTX 3060 12GB, €257 (used (median of sold listings))
Q4, slow, partly in system RAM: Nothing on our list of 29 machines fits this quant.