Fabio Matricardi's ThinkPad 780M CPU AI lab

The two laptops

He uses a local model every day, in llama.cpp, for coding with opencode and for personal documents. The workplace Dell is for proprietary data he will not send out. The ThinkPad is a Lenovo with an AMD Radeon 780M and 32 GB of unified RAM. The Dell is an i5 with an Intel Ultra iGPU and 32 GB of unified RAM.

Mainstream llama.cpp runs Ternary-Bonsai-2-4B-Q2_0.gguf. A PrismML fork of llama.cpp, with MTP heads, runs Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf. Mainstream llama.cpp also runs Qwen3.5-4B-OnlyText-GGUF at Q6_K (3.56 GB), and that file includes MTP heads. The fourth daily file is NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q5_K_XL.gguf. The PrismML fork is what he uses for opencode and for private documents when the model has MTP heads.

Why the quant stays at Q6

For opencode agent-harness runs he never goes below Q6. Q5_K_M is the occasional exception. Weaker quants drift on structured output and on tool calls. The model starts repeating the same reasoning for many turns, or the whole run stops with no alert. Those failures show up inside the agent harness, so he does not have a single pass-or-fail prompt to hand over.

An AMD iGPU is useful here, and ordinary consumer hardware runs out of room above about 8B. An MoE might still fit, and the RAM cost at Q5_K_M is already high. When a job is past that local ceiling he uses a Gemini Plus subscription at $4 per month, for the heavy jobs and for when his free opencode credits run out. He would rather do that than rent a GPU.

The speeds he wants at 40,000 tokens

He looks on Reddit, and sometimes on X, and he does not trust a single first-call number. The speed of the first call and the speed once 30,000 tokens are in the KV cache are different. His work is mostly opencode, so he wants prefill above 100 tokens per second and generation above 25 tokens per second at 40,000 tokens of KV cache. The figures he would call a good fit are prefill above 200 tokens per second and generation above 50 tokens per second, at that same cache size.

The next machine

With a larger budget he would buy a Mac, and it would be a desktop. A machine for local models should stay on, and it should run the models, the guardrails, and the gateway. On his current budget he would look for a mini PC with a strong AMD iGPU and enough CPU cores to run an embedding model as well.

What would make a speed figure trustworthy

A plot against context length, and the settings that go with it: KV cache quantization, context length, and max new tokens. A single tokens-per-second figure without the context length is the number he already does not trust.

What he wishes a hardware site told him

Engine settings, especially for CPU. How many threads to give the model, and why. How many threads to leave free so the laptop is still usable. He likes that a CPU can still do useful work, and outside llama.cpp the options are thin. A context-length throughput curve, and guidance on how to split CPU threads, is what he asked for.