Blackwell quantisation on my own card

Quantising a small model down from FP16 to Q4_K_M on my RTX 5070 Ti more than doubled throughput and cut VRAM by two thirds, with output that stayed 98% similar to the full-precision baseline by embedding distance. Both registered gates passed. The interesting part turned out to be where the remaining 2% actually lived.

Key takeaways

• Q4_K_M reached 258.0 tokens/second against 116.5 for FP16 on the same model and prompts, a 2.22x speedup against a registered gate of 1.3x.
• VRAM dropped monotonically with precision: 6,895 MiB at FP16, 4,019 MiB at Q8_0, 2,594 MiB at Q4_K_M, on the same 16 GB card.
• Average embedding similarity between Q4_K_M and FP16 output stayed at 0.9835 across 20 fixed prompts, against a registered floor of 0.90.
• The lowest-similarity cases were not errors: reading the raw text showed paraphrase on open-ended questions, not factual drift, while every closed factual prompt scored a perfect 1.0000 at every quantisation level.
• This measures GGUF quantisation through Ollama, not the NVFP4 format a related paper tested on the same GPU family; the two are different schemes, and the paper is context here, not a target to reproduce.

Why measure this myself instead of trusting the paper?

A 2026 paper on consumer Blackwell GPUs reports 1.6x throughput for NVFP4 quantisation over BF16, with 2 to 4% quality loss, tested on the RTX 5060 Ti, 5070 Ti and 5090. I own one of those cards, which made the paper a natural registered comparison rather than a citation to take on faith, following the same discipline described in why I pre-register a kill-gate for every experiment: the thresholds below were fixed before any model ran. The catch is that NVFP4 runs through TensorRT-LLM, a stack built for exactly that format; what I actually run day to day is Ollama, which serves GGUF K-quants through llama.cpp. Different scheme, different software, and no honest way to expect the numbers to match.

So the registered question was narrower than "can I reproduce the paper": does the same shape of trade-off, lower precision buying speed and memory at some quality cost, hold for the quantisation format most people on consumer hardware actually use. I picked a small model, llama3.2:3b-instruct, deliberately smaller than anything in the paper, so all three precisions fit comfortably in 16 GB and the comparison measured the trade-off itself rather than whether anything fit at all.

What happened to throughput, memory and output quality?

Every metric moved the same direction as precision dropped, and moved together. Throughput: 116.5 tokens per second at FP16, 187.0 at Q8_0, 258.0 at Q4_K_M, a 2.22x gain at the lowest precision against a registered floor of 1.3x. VRAM at load: 6,895 MiB, 4,019 MiB, 2,594 MiB across the same three steps. Neither curve reversed at any point, which is a cleaner result than I expected going in; I had assumed at least one metric would show a step where a lower precision briefly cost more before paying off, and none did. Full numbers, including wall-clock time for the twenty-prompt run at each precision, sit in the frozen plan and results in the public experiment repository.

Quality held up by the same measure. I embedded every response with a local embedding model and computed cosine similarity between each quantised output and its FP16 counterpart across 20 fixed prompts, generated at temperature zero for reproducibility. Q8_0 averaged 0.9984 similarity to FP16, close enough to call the two outputs interchangeable. Q4_K_M averaged 0.9835, comfortably above the registered floor of 0.90, though with more visible spread across individual prompts than Q8_0 showed.

Where did the quality difference actually come from?

Averages hide the interesting part. The two lowest Q4_K_M scores belonged to open-ended prompts, a two-sentence plot summary and a one-sentence technical definition, not to any of the closed factual questions in the set. Reading the raw text answered the question a similarity score alone cannot: both quantised and full-precision answers stated the same facts in different words, not different facts. Meanwhile every closed factual prompt, a capital city, a square root, a chemical symbol, scored a perfect 1.0000 at every precision level tested, because a correct short answer has almost no room to be phrased two different ways.

That split matters for reading the headline number honestly. A 0.9835 average is not "2% of answers were wrong at random"; it is closer to "open-ended answers get reworded, closed ones do not," which is a much weaker claim about the cost of quantisation than the raw percentage suggests on its own. I would not have caught this by trusting the average. The lesson generalises past this one experiment: an aggregate similarity score is a screening tool, not a verdict, and the prompts sitting at the low end of it are the ones worth reading by hand before writing anything down.

FAQ

Does this mean Q4 quantisation is basically free on consumer hardware?

For a 3B instruction model on a 16 GB card, on short factual and short explanatory prompts, the registered numbers say yes: large speed and memory gains, similarity comfortably above the floor I set. That does not generalise automatically to larger models, longer generations, or tasks with less room for legitimate paraphrase; it is one measured point, not a universal rule.

Why not test NVFP4 directly for a real comparison?

NVFP4 needs TensorRT-LLM, a different serving stack from the llama.cpp backend behind Ollama, and setting that stack up was out of scope for a same-week experiment on my own hardware. The paper's numbers stand as context for what a more specialised stack can do; this experiment measures what the tool most people actually reach for does on the same GPU family.

Why pick such a small model instead of something closer to production use?

A 3B model lets FP16, Q8_0 and Q4_K_M all fit in 16 GB without any of them straining the card, which isolates the precision-versus-speed question from the separate question of what fits at all. That second question, whether a larger model fits at a given precision on this exact card, is real and unanswered here; it belongs to its own experiment.

References

• Jonathan Knoop, Hendrik Holtmann. Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs, 2026. arxiv.org/abs/2601.09527
• Ruibo Fan, Xiangrui Yu, Xinglin Pan, Zeyu Li, Weile Luo, Qiang Wang, Wei Wang, Xiaowen Chu. ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression, 2026. arxiv.org/abs/2603.17435