You are a competent systems engineer. You have been asked to design a single-GPU transformer inference engine that executes prefill and inference on contexts that will not fit in VRAM at speeds that rule out CPU processing. It must not use approximate attention: every token of the context takes part in every attention step, with no selection, sparsity, or dropping of tokens. How would you build it?
Someone already has. It's called StreamingKV.
64,000 tokens on a 16 GB RTX 5080, all 6 facts found, and long answers 5 to 7 times as fast as llama.cpp. A document of 63,975 tokens (a token is a word or part of a word) was read on one GeForce RTX 5080 (16 GB, PCIe gen5 x16, power limit 300 W), a card with less memory than the model alone needs, and the answers gave back all 6 facts we had planted in it. Long answers came at 5.1 to 6.7 tokens per second; llama.cpp, on the same machine with the same model, gave 1.00. On a 24 GB RTX 5090 Laptop GPU, answers came at about 12 to 16 tokens per second against llama.cpp's 1.1, and a question answered in 30 tokens took about 3 seconds from start to finish against llama.cpp's 27. With the 24 GB card the whole model is on the card; with the 16 GB RTX 5080, about 40% of it (7-8 GB) streams from system memory. At 64K tokens, most of the document's working memory (the KV cache) streams from system memory on the laptop, and all of it on the 5080. Measured with the October 25, 2026 release of the free edition of StreamingKV, running Seed-OSS-36B-Instruct. The RTX 5080 machine has a Ryzen 9 9900X3D processor and 32 GB of DDR5-6000 memory. Both programs kept the document's working memory (the KV cache) at 4 bits. llama.cpp was told the document's size in advance, and then put 36 of the model's 65 layers on the card and ran the rest on the processor; when it was not told, it could not read the document.
Early access is limited to 25 copies, for United States citizens and lawful permanent residents, 18 or older, who live in and are in the United States. The request form closes when they have all been given out.
Personal use or evaluation only. No commercial use, and no serving other users. Read the full text before you ask for a copy: StreamingKV Free Edition License.