When does sparse decoding pay off on consumer NVIDIA GPUs?
Sparse Decode Bench is the first forthcoming research project from Controra Research. It asks when sparse decoding pays off for ordinary dense language models in llama.cpp on consumer NVIDIA GPUs, weighing page selection overhead, decoding speed and model quality.
Method
During token decoding, the method under study keeps selected pages of the key/value (KV) cache: the initial context, the recent context and further pages chosen by a score. Nothing is trained. The project compares sparse decoding with dense decoding, measures end to end performance and the cost of page selection, and uses a cost model to predict when sparse decoding pays off.
Scope and limits
The current scope is llama.cpp CUDA, single sequence decoding, consumer NVIDIA RTX 4090 and RTX 3090 GPUs, and the Llama 3.x and Qwen3 model families.
Prefill, the step that processes the prompt, stays dense and unchanged. Page selection applies only to decoding, which generates the next output tokens. Next token divergence alone does not establish task quality.
Methods, code and benchmark notes are being prepared for release.