estimate, except measured file sizes

Can I run Llama 3.3 70B locally?

70.6B parameters. Parameter count is an estimate.

Licence:

not recorded

Context:

131,072 tokens

Modality:

not recorded

Released:

not recorded

Q4 means Q4_K_M. The 0.60 formula is about 42 GB of weights. A bartowski Llama-3.3-70B-Instruct Q4_K_M file is 42.52 GB (42,520,398,816 bytes). A single 24 GB or 32 GB card is not enough.

· Architecture source · Model page

Memory per quant

KV is at 8,192 tokens. Overhead is 0.8 GB. Both are estimates. A measured file size is labelled. The last column is the cheapest machine that fits the total fully.

Cheapest rig that fits

Q4, full: Mac mini M4 Pro 64GB, €1,890 (estimate)

Q4, slow, partly in system RAM: Budget PC with RTX 3060 12GB, slow, partly in system RAM, €1,357 (estimate)

On a specific GPU

• Llama 3.3 70B on RTX 3060 12GB
• Llama 3.3 70B on RTX 4060 Ti 16GB
• Llama 3.3 70B on RTX 3090
• Llama 3.3 70B on RTX 4090
• Llama 3.3 70B on RTX 5080
• Llama 3.3 70B on RTX 5090
• Llama 3.3 70B on RX 7900 XTX
• Llama 3.3 70B on Dual RTX 3090
• Llama 3.3 70B on Dual RTX 4090