What fits in 24GB VRAM? Check the quant, not the label

A 2-bit build is not a 24 GB promise

NVIDIA's Nemotron 3.5 Bolt 74B-A9B ships an official pack aimed at 24 GB. Meta Muse Spark Open also ships a 2-bit GGUF. This catalog's measured ggml-org Q2_K file for Muse Spark Open is 304.08 GB. ggml-org Muse Spark Open GGUF That file is far larger than one 24 GB card. The label on the quant does not tell you the fit. The file size does, and then only after you add cache and overhead.

Nemotron's own 24 GB budget

NVIDIA's technical report says the listed Bolt weights occupy 22.01 GB. Cache and activations add 0.87 GB in NVIDIA's estimate for batch size 1, a context of 131,072 tokens, and a Mamba prefill chunk of 8,192. The drafter they include uses another 0.96 GB. NVIDIA adds that up to 23.83 GB of a 24 GB target, with 0.17 GB left, and says the vision encoder and projector are excluded. Those gigabytes are NVIDIA's estimate. They are not a file this site measured, so they are not in the table below. NVIDIA technical report, 15 October 2026 The walkthrough of that budget is Nemotron 3.5 Bolt hardware requirements.

This site's formula subtracts 0.6 GB per discrete GPU. On the catalog's 24 GB cards that leaves 23.4 GB usable. The GeForce RTX 3090 and the GeForce RTX 4090 are both 24 GB cards in this catalog, so they share that usable figure. The formula's overhead is 0.8 GB. Both numbers are estimates.

Qwen3.8-27B: the quant decides the headroom

Qwen3.8-27B is a dense vision-language model. The card's headline is 27B. Hugging Face safetensors metadata lists 27,781,427,952 parameters, stored here as 27.78B. Apache-2.0. The config has 64 layers, 48 of them linear attention and 16 full attention, 4 KV heads, and head dim 256. Max position embeddings are 262,144. Qwen3.8-27B model card and config.json.

The KV term uses those 16 full-attention layers only. Linear-attention layers add no token KV in this formula. At 8,192 tokens that KV estimate is 0.54 GB. At 32,768 tokens it is 2.15 GB. Overhead stays 0.80 GB.

Unsloth's UD-Q4_K_M file is 16.46 GB, measured. The formula total at 8,192 tokens is 17.80 GB (estimate). Spare room under 23.4 GB usable is about 5.60 GB, so the table says yes. The same file at 32,768 tokens is 19.41 GB (estimate). Spare room is about 3.99 GB, under 4 GB, so the same file is tight once the context grows. unsloth/Qwen3.8-27B-GGUF.

UD-Q6_K is 21.98 GB, measured. The formula total at 8,192 tokens is 23.32 GB (estimate), about 0.08 GB under the usable figure. That is a fit, and it is tight. At 32,768 tokens the same file is 24.93 GB (estimate), which is above 23.4 GB usable. Q8_0 is 29.05 GB, measured. The formula total at 8,192 tokens is 30.39 GB (estimate). That does not fit one 24 GB card. The vision projector is a separate file: mmproj-F16 on that repo is 0.93 GB (927,607,488 bytes, 19 Oct 2026), and it is not inside these totals.

A community results page measured one RTX 3090, the same 4-bit checkpoint, and a ~1K-token prompt. The canonical single-GPU column is the mean of five runs: 512-token greedy completions, reasoning disabled, 250 W power cap, the other replica stopped. llama.cpp out of the box is 27.6 tokens per second. llama.cpp tuned, with flash attention, q8 KV and MTP, is 42.6. Stock vLLM with MTP k=4 is 85.6. Patched vLLM with MTP k=4 is 167.8. Patched vLLM with DFlash2 k=7 is 212.1 tokens per second. Same card, software only. This site did not run that test. The page dates the single-replica correction 31 August, and results.html was last committed on 18 September 2026. Qwen3.8-27B on RTX 3090, results Hubs: Can I run Qwen3.8-27B? and the model page.

Qwen3.6-35B-A3B: a MoE file that can still fit

Qwen3.6-35B-A3B is a mixture-of-experts model. The card says 35B total and 3B activated. Safetensors metadata lists 35,951,822,704 parameters, stored here as 35.95B. VRAM math uses the total. Apache-2.0. The config has 40 layers, 30 linear attention and 10 full attention, 2 KV heads, head dim 256, 256 experts and 8 routed per token. The card adds one shared expert. Qwen3.6-35B-A3B model card and config.json.

Measured Unsloth files, decimal GB, 19 Oct 2026: UD-Q3_K_XL 16.85, UD-IQ4_XS 17.73, MXFP4_MOE 21.71, UD-Q4_K_M 22.13, UD-Q4_K_XL 22.36, Q8_0 36.90 (36,903,140,320 bytes). The KV estimate at 8,192 tokens is 0.17 GB, from the 10 full-attention layers only. UD-Q4_K_M plus that cache and the 0.80 GB overhead is 23.10 GB (estimate). That still fits 23.4 GB usable, with little room left, so the table says tight. Q8_0 does not. mmproj-F16 on that repo is 0.90 GB (899,283,680 bytes) and is not in the total. No speed figure is stored. Speed is not estimated for this mixture-of-experts model. unsloth/Qwen3.6-35B-A3B-GGUF. Hub: Can I run Qwen3.6-35B-A3B?.

Qwen3.8-Flash-Next does not fit one 24 GB card

Qwen3.8-Flash-Next is a much larger mixture-of-experts model. The card says 125B with 6B activated, plus 51B n-gram embedding and 4B MTP. Safetensors metadata lists 179,999,981,459 parameters, stored here as 180B total. VRAM math uses the total. The licence file is the Qwen Community License 1.0. Hugging Face metadata records license: other and license_name: qwen-community-1.0. Qwen3.8-Flash-Next model card.

The smallest Unsloth file in this catalog, UD-IQ4_XS, is 93.68 GB, shards summed on 19 Oct 2026. UD-Q4_K_XL is 111.33 GB. Q8_0 is 188.23 GB. The MTP drafter MTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguf is 4.14 GB and is not included in those totals. mmproj-F16 is 0.90 GB (904,004,000 bytes), also separate. Each of those weight files is already larger than a 24 GB card. unsloth/Qwen3.8-Flash-Next-GGUF.

The 12 full-attention layers are Qwen Sparse Attention. The card's budget is 2,048 tokens, and the config's indexer_budget is 2048. This catalog does not compute that cache, so the table adds only the 0.8 GB overhead estimate. The file size alone answers the 24 GB question. No speed figure is stored.

Strata's README says the graphics card keeps the experts that are used most often and RAM holds all of them. That is expert residency in system RAM, in Strata, not a claim that the GGUF fits in 24 GB of VRAM. Strata. Releases: Strata releases. Mainline llama.cpp pull requests #27861, #30594 and #30637 were still open on 19 Oct 2026. This page does not treat a GPU expert cache as available in mainline llama.cpp. Hub: Can I run Qwen3.8-Flash-Next?.

Measured files against one 24 GB card

File GB is the measured GGUF size (bytes / 1e9, two decimals). Total at 8K adds this site's FP16 KV estimate and 0.8 GB overhead. yes means the total fits in 23.4 GB usable with at least 4 GB spare. tight means it fits and the spare is under 4 GB. no means the total is above that usable figure. The 4 GB line is the same headroom rule the calculator uses. Qwen3.8-Flash-Next rows do not include a KV cache, because that cache is not modelled. The hardware pages' Q4 column for these models uses the measured Q4-class file. Their Q5, Q8 and FP16 columns are the formula rates, not these GGUF listings.

What to check before you download

• The file size on the repo, in decimal GB. Sharded quants need every shard added together. A 2-bit or 4-bit name is not the size.
• The context you will actually use. On Qwen3.8-27B, UD-Q4_K_M has room at 8,192 tokens and is tight at 32,768 tokens in this formula. UD-Q6_K flips from tight to no across that same step.
• The KV quant. This formula keeps KV in FP16. The 3090 study's tuned llama.cpp figure uses q8 KV, which is a smaller cache than the one in this table. A smaller KV quant is not already included.
• A vision projector, if you want images. Measured mmproj-F16 files on 19 Oct 2026: 0.93 GB for Qwen3.8-27B, 0.90 GB for Qwen3.8-Flash-Next, and 0.90 GB for Qwen3.6-35B-A3B, each on the Unsloth GGUF repo linked above. Those files are not inside the table.
• Other programs on the GPU. The formula already subtracts 0.6 GB per discrete GPU. A browser, a desktop compositor, or a second model sits on top of that reserve.

Where to look next

• GeForce RTX 3090 24GB and GeForce RTX 4090 24GB
• RTX 3090 vs 4090 for local LLMs
• What the quant names mean
• VRAM calculator
• Qwen3.8-27B hub, Qwen3.8-Flash-Next hub, Qwen3.6-35B-A3B hub

Open the dense 27B model on the calculator. Q4 there is the measured UD-Q4_K_M file, at 8,192 tokens. The result is an estimate: Qwen3.8-27B Q4.

Open the calculator