Roshan George's RTX A6000 48 GB shared vLLM server

The server

The machine is an AMD EPYC 7402P (24 cores) on a Gigabyte MZ32-AR0 board with 256 GB of DDR4-2400. The GPU is one used NVIDIA RTX A6000 48 GB, Ampere, compute capability sm_86. At 300 W in a cube case the card hit 87 °C, so he caps it at 250 W.

vLLM runs in rootless Podman. The same card also decodes NVR camera video and serves embeddings and speech. That leaves about 35 GB of the 48 GB for vLLM. He says planning around that shared memory has mattered more than raw speed.

What the card runs

Today the model is a fine-tuned Gemma 4 26B A4B, a mixture of experts, quantized to GPTQ W4A16 with group size 64. It is a fast decision model for a browser agent. The agent answers multiple-choice questions from the logprobs of one forward pass, instead of generating text.

Before that he ran Qwen3.8-27B in AWQ W4A16 with speculative decoding. MTP-2 gave about 60 tokens per second single-stream. The DFlash2 drafter gave 100 to 129 tokens per second on code, and 53 to 66 tokens per second on prose. Batched, with speculation off, the same setup peaked around 690 tokens per second at 16 concurrent requests.

On this Ampere card (sm_86) he always uses INT4 W4A16 through Marlin kernels. FP8 checkpoints either fail to load, because they need sm_89, or run weight-only. Weight-only FP8 measured 35 percent slower than INT4 on the same model. NVFP4 is only a fallback.

The second A6000

He already owns a second RTX A6000 48 GB. The upgrade cost is $0. He will install it so the workloads split: vLLM alone on one card, and NVR, embeddings, and speech on the other.

In a follow-up on 26 Oct 2026 he added that single-GPU inference on an RTX A6000 makes sense when the card is paired over NVLink. For pure compute he would rather use an unlocked CMP 170 HX. On this home box a CMP 170 HX has no NVDEC, which would break camera decode. He also turned down a 48 GB RTX 4090 mod, because 48 GB stays 48 GB and does not hold a larger model. His larger spend is a separate 4-GPU RTX PRO 6000 datacenter build.

Where he looks before buying

He checks vLLM GitHub issues and pull requests, llama.cpp discussions, model cards, and the checkpoint's safetensors index, so he can see what is actually quantized and what is left in BF16.

What is usually missing is a number that states the engine, the quant format, the context length, the concurrency, and the power limit. llama.cpp and vLLM with MTP on the same card can differ by 2× or more. Memory figures should include the KV cache per token at the context length he actually runs.

What would make a speed figure trustworthy

The exact engine version and the launch flags. The quant repo and its revision. Whether speculative decoding was on, and what kind of prompt it was: code, prose, and repetitive text can differ by 2× once speculation is on. Single-stream and batched numbers reported separately. The power limit and the temperature. A benchmark script he can run.

What he wants on a hardware guide

Which quant formats each GPU generation runs natively, and which it emulates. FP8 on Ampere is weight-only and measured 35 percent slower than INT4 Marlin. Whether a quantized checkpoint actually loads on a given vLLM version, including mismatched ignore lists and MTP heads. Speed when the GPU is shared with other services. Tokens per second split by workload type and by concurrency.