The StreamingKV Free Edition runs a language model over a document longer than the graphics card can hold, by streaming the part that does not fit from the computer's main memory as it is needed. System memory, not graphics memory, sets how much context you can use -- and you do not have to decide how much before you start.
The free edition is source code from which you build the executable yourself, for personal use or evaluation. Early access is limited to 25 copies, for United States citizens and lawful permanent residents, 18 or older, who live in and are in the United States.
We ran StreamingKV and llama.cpp side by side with one script, build/compare.sh, which is part of the download: it has each program read the same documents, asks each the same questions, and times both.
The model is Seed-OSS-36B-Instruct, a 36-billion-parameter model whose file is about 22 GB -- more than a 16 GB card holds. Neither program can keep the whole model on a 16 GB card. llama.cpp computes the layers that do not fit on the processor; StreamingKV keeps them in system memory and copies each to the card as it is needed, so its speed depends on how fast the card's PCIe link is.
So we measured on the two links a current desktop has. We rented three machines in data centers -- one with a PCIe gen4 x16 link, and two with gen5 x16, one for the 16 GB results and one for the whole RTX 5090 -- and ran both programs on each, one at a time, with the card to itself:
-c (told: the document's length plus 556 tokens, room for the question and a 300-token answer; untold: not given) --parallel 1 -fa on -ctk q4_0 -ctv q4_0 -lv 4 (the last only sets how much it logs). No --threads and no --numa: it picked its own thread count -- one per core on the rented servers, 4 on the laptop's 24-core processor -- and ran with no NUMA setting. StreamingKV switches on speculative decoding by itself; llama.cpp ran without its own. We tried llama.cpp's built-in speculation that needs no second model (n-gram drafting, --spec-default) on this model, on an RTX 5090 Laptop GPU with a PCIe gen5 x16 link: on real answers it found nothing to draft, so it answered at the same speed as without it. We are not llama.cpp experts: if you can make it faster on this test, with its own speculative decoding or otherwise, tell us how and we will publish it.Every run wrote a record of exactly what produced it -- the SHA-256 of each program and of the model, the hardware above, and the command line -- and every figure below is copied from those runs' own logs.
Both programs read the same book, A Journey in Other Worlds by John Jacob Astor IV (1894, public domain), and were asked the same questions, word for word.
Six short facts are planted in the book at different depths. It is cut to a series of lengths, listed in the tables below, and each program reads it at every length. The book is cut only between paragraphs, never in the middle of one, so each length comes out a little under the round number it was cut for. After reading it, each program is asked for the six planted facts, then one or more longer questions whose answers are in the book itself.
These are the questions. build/compare.sh asks them itself, in the same words for both programs, and prints every answer. The six planted facts are asked one at a time:
The longer questions about the book:
Both programs are asked the first of these at every length, and all four at the longest. Each answer may run to at most 300 tokens.
llama.cpp has to be told or assume, before it starts, how much context to set memory aside for -- and that has to hold every token it will compute over: the document, the question and the answer, not the document alone. So we ran it two ways:
StreamingKV is started the same way every time. It is never told the size.
| Length (tokens) | StreamingKV, gen4 | llama.cpp, gen4 | StreamingKV, gen5* | llama.cpp, gen5* |
|---|---|---|---|---|
| 19,987 | 3.5 | 1.7 | 7.9 | 2.1 |
| 39,803 | 4.1 | 1.1 | 7.1 | 1.4 |
| 59,873 | 3.7 | 0.72 | 6.2 | 1.1 |
| 79,816 | 2.9 | 0.43 | 6.7 | 0.84 |
| 99,777 | 2.2 | 0.35 | 4.7 | 0.64 |
| Length (tokens) | StreamingKV, gen4 | llama.cpp, gen4 | StreamingKV, gen5* | llama.cpp, gen5* |
|---|---|---|---|---|
| 19,987 | 35.9 | 43.1 | 21.1 | 34.4 |
| 39,803 | 75.9 | 131.4 | 37.8 | 78.7 |
| 59,873 | 134.5 | 292.7 | 67.7 | 147.9 |
| 79,816 | 217.4 | 427.0 | 105.6 | 217.4 |
| 99,777 | 309.3 | 693.3 | 152.8 | 316.6 |
* Simulated 16 GB card: the gen5 columns were measured on an RTX 5090 held to that much free memory. No 16 GB card on a gen5 link was for rent; a real one, such as an RTX 5080, has a slower chip and will likely be slower. We have a real RTX 5080 on order and will replace these columns with its measurements.
Gen4: RTX 5070 Ti, PCIe gen4 x16, two AMD EPYC 7B13. Gen5: RTX 5090 held to 15,841 MiB free, PCIe gen5 x16, two AMD EPYC 9654. llama.cpp release v0.5.0, told the size. Both programs found all six facts at every length, on every card and link. llama.cpp kept 42 of its 65 layers (llama.cpp counts the model's 64 transformer layers plus its output layer) on the card at 19,987 tokens and 32 at 99,777 on gen4, and 41 and 32 on gen5. At 99,777 tokens the answering speed averages all four long answers; at the other lengths it is the first. Each figure is one run: the same rented card varied by about 15% from one run to the next, so neighbouring rows can cross (StreamingKV's 40,000 row above its 20,000 row on gen4, for example). Run on 2026-10-12, StreamingKV from the release this page describes. For the runs with memory held back, compare.sh had one line changed so that it would run beside the program holding the memory; nothing else in it was. StreamingKV switches on speculative decoding by itself once part of the model or the document streams from system memory, and in the runs where we checked, its answers were the same, word for word, as without it; on the whole RTX 5090 nothing streams, and it answered without speculation at every length. The ratios in the text are worked out from the figures in the run records, so dividing the rounded numbers in a table can give a slightly different result.
With a real 16 GB RTX 5070 Ti on a gen4 link, StreamingKV answered 2.1 times as fast as llama.cpp at 20,000 tokens, rising to 6.7 times at 80,000 -- 2.9 tokens per second against 0.43 -- and read 1.2 to 2.2 times as fast.
On a PCIe gen5 x16 link with 16 GB, StreamingKV answered at 7.9 tokens per second at 20,000 tokens, 6.7 at 80,000 and 4.7 at 100,000; llama.cpp, at 2.1, 0.84 and 0.64 -- 3.8 to 8.0 times as fast. From 40,000 tokens on, StreamingKV read in about half llama.cpp's time -- 106 seconds against 217 at 80,000 tokens -- and in 61% of it at 20,000. StreamingKV ran about twice as fast on the gen5 machine as on gen4. Most of that is the link, which the layers that do not fit cross for every token, but not all of it: the gen5 machine also had a faster card (an RTX 5090 held to 16 GB) and newer processors. llama.cpp was faster on the gen5 machine too; its newer processors with faster memory set its speed when layers run on the processor.
A slower link means a slower StreamingKV when the model does not fit on the card. Check yours with the command under What do I need to run it?
With 32 GB the whole model fits on the card, and llama.cpp keeps every one of its 65 layers there at every length we ran. This is the case llama.cpp is built for, and it reads the document faster than StreamingKV. Nothing streams for StreamingKV either, so it answered without speculative decoding here. StreamingKV still answers faster: 2.2 times as fast at 20,000 tokens, rising to 4.7 times at 80,000 and 5.3 times at 100,000. Not told the size, llama.cpp chose a context of 135,424 tokens and read every length.
The RTX 5090 Laptop GPU has 24 GB, a little more than the model's 22 GB file. llama.cpp keeps all 65 of its layers on that card at 20,000 tokens and fewer as the document grows -- 52 at 100,000. StreamingKV answered 1.3 times as fast at 20,000 tokens, 5.4 times at 40,000, 18 times at 80,000 and 23 times at 100,000, and read faster from 60,000 tokens on; at 40,000 the two read in the same time. Not told the size, llama.cpp chose 28,672 tokens and could read only the 20,000-token document.
| Length (tokens) | Read, StreamingKV (s) | Read, llama.cpp (s) | Answer, StreamingKV (tokens/s) | Answer, llama.cpp (tokens/s) | llama.cpp layers on the card (of 65: 64 transformer layers + the output layer) |
|---|---|---|---|---|---|
| 19,987 | 17.5 | 9.4 | 48.4 | 22.1 | 65 of 65 |
| 39,803 | 33.0 | 26.2 | 44.2 | 13.7 | 65 of 65 |
| 59,873 | 62.4 | 52.1 | 40.2 | 10.0 | 65 of 65 |
| 79,816 | 99.8 | 87.0 | 37.0 | 7.8 | 65 of 65 |
| 99,777 | 146.2 | 134.6 | 34.0 | 6.4 | 65 of 65 |
NVIDIA GeForce RTX 5090, the whole card; PCIe gen5 x16 under load; AMD EPYC 9654, rented in a data center. llama.cpp release v0.5.0, told the size. Both programs found all six facts at every length. At 99,777 tokens the answering speed averages all four long answers; at the other lengths it is the first.
| Length (tokens) | Read, StreamingKV (s) | Read, llama.cpp (s) | Answer, StreamingKV (tokens/s) | Answer, llama.cpp (tokens/s) | llama.cpp layers on the card (of 65: 64 transformer layers + the output layer) |
|---|---|---|---|---|---|
| 19,987 | 28.3 | 26.1 | 23.0 | 17.7 | 65 of 65 |
| 39,803 | 77.7 | 78.3 | 19.5 | 3.6 | 62 of 65 |
| 59,873 | 149.6 | 179.8 | 16.1 | 1.1 | 58 of 65 |
| 79,816 | 242.6 | 284.0 | 11.4 | 0.63 | 55 of 65 |
| 99,777 | 357.4 | 453.2 | 9.4 | 0.40 | 52 of 65 |
NVIDIA GeForce RTX 5090 Laptop GPU, 24,463 MiB, the whole card with the laptop's desktop running; PCIe gen5 x16 under load; Intel Core Ultra 9 285HX, 64 GB. Our own machine, not rented. llama.cpp release v0.5.0, told the size. Both programs found all six facts at every length.
MLCommons, the industry group behind the MLPerf benchmarks, sets a speed each user must get from a chat model in its MLPerf Inference benchmark (version 5.0, 2025). In its server scenario -- an online service answering a stream of users -- each user must receive a new token at least every 200 milliseconds: 5 tokens per second, about as fast as an adult reads. (Its stricter interactive scenario asks for 25.) For a single user, on the simulated 16 GB gen5 card, StreamingKV met the server rate up to 80,000 tokens (6.7 tokens per second there) and fell below it at 100,000 (4.7). On a gen4 link it did not meet it: 2 to 4 tokens per second with 16 GB -- slower than reading speed, though llama.cpp on the same card managed 0.35 to 1.7. On a whole RTX 5090 it answered at 34 to 48, and on the RTX 5090 Laptop GPU at 9.4 to 23. llama.cpp met the server rate only on the whole RTX 5090 (6.4 to 22) and, at 20,000 tokens, on the laptop.
Reading time is a different story: at these lengths reading the document takes minutes, for both programs. But it is done once, like opening a file. The chat program included in the download answers as many questions as you like from one read, and a read saved with /save can be loaded again with /load instead of being read again.
Everything above comes from build/compare.sh, which is part of the download. These are the exact runs:
the novel, 20,000 to 100,000 tokens:
bash build/compare.sh --sizes "20000 40000 60000 80000 100000" --llama-ctx told,untold --answer-tokens 300
--answer-tokens 300 matters: without it each answer may run to 2,000 tokens, so the answers, and the speeds, will not match the ones on this page. Our 16 GB gen5 figures come from a 32 GB card with part of its memory held by another program; on a card that really has 16 GB, run the command as it is. The whole series takes about 100 minutes on a 16 GB card on a gen4 link -- most of it llama.cpp's; build/compare.sh --minutes sets a limit.
The script fetches llama.cpp from its own project (release v0.5.0, MIT licensed) and builds it on your machine, prints every answer from both programs, and writes a record of exactly what it ran and on what hardware. You are welcome to publish what you find (see Can I publish what I find? below).
No. With -nkvo the context stays in system memory and llama.cpp moves the attention work to the CPU to meet it; its own source says so (src/llama-graph.cpp). StreamingKV moves the context to the graphics card, a piece at a time as it is needed, and does the attention there. One moves the work to the data; the other moves the data to the work.
Those methods make the context smaller by giving up part of what the model can see. A sliding-window layer attends only to the most recent few thousand tokens and cannot look further back. A linear-attention layer folds its whole history into a summary of fixed size, which by construction cannot keep all of it. Models built this way keep some layers with full attention for exactly that reason, and they hold less context because most of their layers no longer see all of it. That is a trade, not a saving: whether it costs answers from deep in a long document is a question for a planted-fact test like the one above, not for a memory figure.
StreamingKV does not shorten what the model sees. Every layer attends to every token of the context, as the model was trained to. What changes is only where the context is held: on the graphics card as far as it fits, and streamed to the card from system memory for the rest. The free edition includes a small tool, kv_cost, that reads any model and reports how much memory its context needs on your card, and how many of its layers see the whole context rather than a limited window -- so you can check whether a model saves memory by seeing less. The memory figure kv_cost reports is what a program needs if it keeps the whole context on the card. StreamingKV does not need it; it streams whatever does not fit.
The source is confidential, so we do not describe the design here. The question at the top of the home page is the problem it solves; if you would like to try designing a solution yourself, start there.
The model's file is about 22 GB, so with 16 GB neither program can keep the whole model on the card. llama.cpp computes the layers that do not fit on the CPU -- and every token has to pass through every layer. StreamingKV keeps those layers in system memory too, but copies each one to the card when it is needed and computes every layer there, along with the part of the context that does not fit. One moves the work to the data; the other moves the data to the work.
How slow llama.cpp is depends heavily on the processor: the layers it moves to the CPU are computed there, split across its cores, and for every token they must read their share of the model and attend over their share of the context, both held in system memory. The machines we rented have large data-center processors with many memory channels -- two-socket servers with 128 to 192 cores and eight or twelve memory channels per processor -- far more cores and memory channels than a desktop; we did not tune llama.cpp's thread or NUMA settings. How slow StreamingKV is depends instead on the PCIe link, which is why we give both gen4 and gen5.
Yes. With 16 GB on a gen4 link its answering speed was 3.5 tokens per second at 20,000 tokens and 2.2 at 100,000, and a slower PCIe link slows it further. The cost grows with how much of the model and the context has to be streamed; speculative decoding, which it turns on by itself once anything streams, recovers part of it. It stayed 2.1 to 6.7 times as fast as llama.cpp at every length on that card.
Then llama.cpp can hold it -- but only because you set that memory aside before you started, and it stays set aside whether the next document is long or short. StreamingKV does not ask you to decide the size in advance.
Every length has six facts planted in it with known answers, and each program is asked for all six. Both programs found all six in every document they read, on every card and link we ran. The long questions are asked twice -- with nothing loaded, then with the text -- so you can see what the model knows on its own and what it got from the document. And every answer is printed in full when you run it yourself.
Yes. Please include the graphics card, its PCIe generation and lane width, and the processor if llama.cpp ran layers on it: without those a number cannot be compared with anything. If you send your figures to us, we may publish them under "reviewer benchmarks", labeled as numbers we did not run ourselves. We put your name to them only if you say in what you send that we may (License, section 11.2).
See What you need, below.
The free edition runs Seed-OSS-36B-Instruct, from ByteDance's Seed team, on every card. It was trained to read 524,288 tokens, so asking it for a very long context is a fair question about the software rather than about whether the model was ever meant to read that much. Its file is about 22 GB, more than a 16 GB card holds: StreamingKV streams the layers that do not fit. One model is one thing to support well.
This first release is early access. Keeping it to the United States -- citizens and lawful permanent residents who live here -- keeps the launch simple: privacy law differs from country to country, and for a first release we would rather meet one set of requirements well than many at once. The number is limited so that we can solve any setup problems the first reviewers run into before opening it to a larger group. Every request is reviewed by hand. We expect to offer more copies later.
No. It is confidential source code, licensed to you for personal use or evaluation for 60 days from when you accept the license. You build it yourself from that source.
Nothing to us, unless you choose to send a review or a problem report from its menu. (Setting it up does fetch the model, llama.cpp and a few software packages from their own publishers; the privacy notice lists them.) Then it shows you eight values first -- version, graphics card, driver, CUDA, operating system, memory, model file and whether it is built -- and asks whether to include them in the link that opens the form. Enter means no; the form still opens, with nothing filled in. The identifier of your copy is never included.
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv shows what yours gets.The setup script checks your machine, installs only what is missing (it asks first), builds, and offers a short check that asks the six planted questions.
Personal use or evaluation only; no commercial use, and no serving other users. One computer, one session at a time. The source is confidential. The license lasts 60 days from when you accept it. Read the full text before you ask: https://streamingkv.com/license.txt
Request a copy: https://forms.fillout.com/t/7BeTNjFiDTus -- details at https://streamingkv.com. Requests are reviewed by hand. An approved request is answered with a single-use download link that works once and expires after 7 days. No suitable graphics card? Rent one on RunPod, step by step -- and read that page before you use the one-time download link.
Once you have run it, tell us whether you believed it -- and if not, what would have convinced you: https://forms.fillout.com/t/eb8zwYgAcKus. Problems go here: https://forms.fillout.com/t/iEfMtXYogaus. The free edition comes with no guaranteed response.
StreamingKV LLC, Dike, Texas. Patent pending.