You are a competent systems engineer. You have been asked to design a single-GPU transformer inference engine that executes prefill and inference on contexts that will not fit in VRAM at speeds that rule out CPU processing. It must not use approximate attention: every token of the context takes part in every attention step, with no selection, sparsity, or dropping of tokens. How would you build it?
Someone already has. It's called StreamingKV.
An LLM inference engine that runs contexts longer than VRAM. Patent pending.
Do you want to spend $6,150 or more on an RTX 5090 (Amazon, October 12) -- if you can even find one -- to run ByteDance's Seed-OSS-36B-Instruct, a 36-billion-parameter model whose Q4_K_M file alone is 22 GB? With an 80,000-token document loaded, llama.cpp v0.5.0 on that card, with all of the model's layers on the card (its 64 transformer layers and its output layer, which llama.cpp counts as 65), answers at 7.8 tokens per second.
With StreamingKV you decide how much to spend for how fast you want to work. Tokens per second while answering, with an 80,000-token document loaded, measured with the free edition:
Whole card, 32 GB: nothing streams here; these rows compare the two programs' answering speed when everything fits, both at a 4-bit KV cache.
| Card | Price | Program | Tokens per second | Tokens per second per $1,000 of price |
|---|---|---|---|---|
| RTX 5090, 32 GB | $6,150 or more | llama.cpp | 7.8 | 1.3 |
| RTX 5090, 32 GB | $6,150 or more | StreamingKV | 37 | 6.0 |
Smaller cards, 16 and 24 GB: the model and the document do not all fit on the card, so llama.cpp runs part of the model on the processor and StreamingKV streams what does not fit.
| Card | Price | Program | Tokens per second | Tokens per second per $1,000 of price |
|---|---|---|---|---|
| RTX 5080, 16 GB* | $1,700, card only | StreamingKV | 6.7* | 4.0* |
| RTX 5080, 16 GB* | the same card | llama.cpp | 0.84* | 0.50* |
| RTX 5090 Laptop GPU, 24 GB | $3,500 or more for the whole laptop | StreamingKV | 11.4 | 3.3 |
| RTX 5090 Laptop GPU, 24 GB | the same laptop | llama.cpp | 0.63 | 0.18 |
* Simulated, a best case: measured on an RTX 5090 held to 16 GB of free memory, on a PCIe gen5 link. A real 5080 has a slower chip and will likely be slower, and it needs a PCIe gen5 motherboard to come close. We have a real RTX 5080 on order and will replace these rows with its measurements.
Both programs read the same model file and found all six facts we planted in the document, on every card. llama.cpp was given flash attention and a 4-bit KV cache (q4_0) -- the same 4 bits StreamingKV keeps -- and placed its layers by its own default (what fits on the card, the rest on the processor); its full flags are on the results page. On the whole RTX 5090 nothing has to stream, so StreamingKV answered there without speculative decoding; it switches speculation on by itself only when part of the model or the document streams, and in the runs where we checked, its answers were the same, word for word, as without it. We are not llama.cpp experts: if you can make it faster on this test -- with its own speculative decoding, say -- tell us how and we will publish it.
Already have a 16 GB card in an older machine? On a real RTX 5070 Ti with a PCIe gen4 x16 link, in a rented data-center server, StreamingKV answered at 2.9 tokens per second -- 6.7 times as fast as llama.cpp, which managed 0.43. That is less than half the gen5 figure: a gen4 link carries half as much, and the gen5 machines also had a faster card and newer processors. That is why we recommend a PCIe gen5 card with 16 GB or more, or a 24 GB card on PCIe gen4 (What you need).
Prices: Amazon for the cards (card only; the computer is extra) and Micro Center for the whole laptop, October 12, 2026. The laptop is our own. The RTX 5090 rows: a rented desktop RTX 5090, the whole card, PCIe gen5 x16, AMD EPYC 9654; the novel at 79,816 tokens, the first long answer. To run the same test on your own card, use the command under Check it yourself. Full results: every card, every length, reading times, and how we ran it.
StreamingKV keeps on the GPU as much as fits there -- the model's layers and its KV cache -- and streams the rest through the card from system memory as it is needed. Either the model or the KV cache, or both, may be streamed, and every layer still runs on the graphics card. So a context can be larger than the card, and so can the model: this edition's model, about 22 GB, runs on a 16 GB card. System RAM bounds how much context you can hold.
The free edition builds a cache from a document, saves it, reloads it, answers questions against it, and keeps a model resident in an interactive session. It runs one session per machine. It is supplied as source code, and you build it yourself.
Tokens per second while answering, with a 16 GB card on a PCIe gen4 x16 link:
| Length (tokens) | StreamingKV | llama.cpp |
|---|---|---|
| 19,987 | 3.5 | 1.7 |
| 39,803 | 4.1 | 1.1 |
| 59,873 | 3.7 | 0.72 |
| 79,816 | 2.9 | 0.43 |
| 99,777 | 2.2 | 0.35 |
An NVIDIA RTX 5070 Ti (16 GB), with two AMD EPYC 7B13 processors, rented in a data center. llama.cpp release v0.5.0, told the document's size in advance. Both programs keep the KV cache at 4 bits. The 99,777-token row averages four long answers; the others, one. Each figure is one run, and the same rented card varied by about 15% from one run to the next, so neighbouring rows can cross. Full results: every card, gen5 beside gen4, reading times, and how to run the same test on your own card.
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv while the card is busy
(an idle card drops to a lower speed).Read the full text before you ask for a copy: StreamingKV Free Edition License. For commercial use, see commercial licensing.
Early access is limited to 25 copies, for United States citizens and lawful permanent residents, 18 or older, who live in and are in the United States. The request form closes when they have all been given out.
tar xzf streamingkv-free-*.tar.gz cd streamingkv-free-*/ bash build/setup.shIt checks your machine, installs only what is missing (it asks first), builds, and offers a short check. docs/README.md in the folder explains every step.
Each copy is marked with identifiers linked to its acceptance. See the privacy notice for what we record and why.
Report a problem, or tell us what you found -- your judgment of it is the more useful of the two. The free edition comes with no guaranteed response.
StreamingKV LLC, Dike, Texas. Patent pending. Privacy notice.