StreamingKV home RUNPOD

StreamingKV Free Edition -- renting a GPU machine

This file is for a reviewer who does not own a suitable graphics card and has never rented a machine by the hour. It walks through three providers, RunPod, Amazon Web Services (EC2) and Vast.ai, from making the account to deleting the machine, and says what to look for and what to avoid at each. Read the README's WHAT YOU NEED first: everything there applies to a rented machine too, and the steps below refer to the README's other sections by name.

Prices and screen labels below were read from each provider's own pages in October 2026. Prices change; treat every figure as approximate and read the current one off the provider's page before you commit. Where a label on the screen differs from what is written here, the provider has renamed it.

THE RULES THAT COME FIRST

  1. THE SOURCE IS CONFIDENTIAL (LICENSE, Section 3), and wherever you put it, you remain responsible for it. The steps below copy the release file to the rented machine and build it there, which needs nothing on your own computer but a terminal. The source then sits on the rented machine's disk until you delete the machine, so delete it as soon as you are done (rule 3). If you happen to have a Linux computer with Docker, or Ubuntu 24.04 with the CUDA toolkit, you can instead build at home and send only the program (OPTIONAL: BUILD AT HOME, below); that is a choice, not a requirement.

  2. RENT ONLY A MACHINE THE PROVIDER RUNS IN A DATA CENTER. A rented machine is acceptable so long as it is not someone's personal machine to which they have root access. The three providers below each have a tier that meets this and, for two of them, a tier that does not:

    RunPod   -- Secure Cloud only. Never Community Cloud.
AWS EC2  -- every instance is Amazon's own; nothing to avoid.
Vast.ai  -- only offers under the "Secure Cloud" filter, with
            the blue datacenter label. Nothing else.

    The disk of a machine in someone's house is theirs to read.

  3. DELETE THE MACHINE WHEN YOU ARE DONE, so that its disk is destroyed and the bill stops. On every provider there is a "stop" that is NOT this: a stopped machine keeps its disk, keeps the source if it is there, and in most cases keeps billing for the disk. Use Terminate (RunPod, AWS) or Destroy (Vast.ai), then look at the provider's list of your machines and check that nothing of yours is listed.
  4. STAY IN THE UNITED STATES. The license is for a person in the US; rent the machine in a US region too, so the question of where your copy is never arises. Each section below says how to choose.

One computer, one session: the license lets one copy run on one computer. While your copy is on the rented machine, that is its computer; do not run another copy at home at the same time.

RUNPOD, ALL IN YOUR BROWSER: docs/RUNPOD.md

For someone who has never rented a machine and does not want to learn about ssh keys, docs/RUNPOD.md walks through RunPod from making the account to deleting the machine, all in a web browser, with nothing installed on your own computer. We walked those steps ourselves on a rented machine. Read THE RULES THAT COME FIRST above, then follow it. The rest of this file is the detail behind it, and the other providers.

WHAT TO RENT

The same things WHAT YOU NEED asks of your own machine, in the words the providers use:

WHICH PROVIDER

RunPod (Secure Cloud) is the easiest first rental: the account is a card and prepaid credit, a machine is running within minutes, there is no approval to wait for, and the GPU prices are the lowest of the three for the cards we recommend (about $0.27 to $1.19 an hour, below). Its machine is a container, not a whole computer: you are root, there is no sudo, and the NVIDIA driver belongs to the host, so build/setup.sh (which installs a driver and reboots) does not apply; the steps below use the README's INSTALLING BY HAND lines instead. The exact commands in the RunPod section follow RunPod's documentation as of October 2026; this version of them has not yet been run end to end by us.

AWS EC2 gives you a whole Ubuntu computer, so build/setup.sh works there exactly as on your own machine, and Amazon runs every instance itself. It costs more per hour than RunPod for a comparable card (about $1 to $2.25), the console has far more on its screens than you need, and A NEW ACCOUNT CANNOT START A GPU INSTANCE until a quota request is approved, which can take hours or days. If you already have an AWS account with GPU quota, it is the most comfortable of the three.

Vast.ai is a marketplace: the prices are set by the people who own the machines and are often the lowest of all, but most of the offers are exactly what rule 2 excludes. Only the "Secure Cloud" filter (offers with the blue datacenter label) qualifies, and it is a small part of the marketplace, so a qualifying US offer with the card you want may not be there on the day. If it is not, use RunPod or AWS; do not relax the filter.

In short: RunPod unless you have a reason to prefer another; AWS if you have an account with GPU quota already or want a whole machine; Vast.ai only with the filter on and only if a qualifying offer exists.

WHAT IT COSTS TO FORGET

Billing runs from the moment the machine starts until you delete it, whether or not you are using it, and a 24 GB card is roughly a dollar an hour: about $20 to $25 a day, $150 to $170 a week, for a machine nobody is logged into. Decide before you start how you will be reminded to delete it. Stopped is not deleted: a stopped machine still bills its disk on all three providers (and on RunPod a stopped Pod loses its container disk anyway, so there is nothing to come back to). All three bill against a card or prepaid credit; RunPod and Vast.ai stop the machine when prepaid credit runs out, AWS keeps charging the card. Delete the machine the same day you finish, and check the list.

PRIVACY

The provider can read the disk of any machine it runs; that is what rule 2 narrows to a company in a data center rather than a person at home. Put nothing on the machine you would not hand to the provider: your own long documents stay on your computer unless they are public text, and your ssh private key never leaves your computer (only the .pub half is given to the provider). Everything on the machine is gone when you delete it; copy out what you want to keep first (the results file, the terminal's text).

WHAT YOU NEED ON YOUR OWN COMPUTER

A terminal with ssh, scp and (for RunPod's command path) curl and python3. Ubuntu, any other Linux, and macOS have them; on Windows 10 or 11, PowerShell has ssh and scp built in, or use Ubuntu under WSL. CHECK:

command -v ssh scp curl python3

Each must print a path. If one prints nothing, on Ubuntu:

sudo apt-get install -y openssh-client curl python3

A GPU is not needed on your own computer, and neither is Ubuntu or any Linux.

If you would rather not use ssh or scp from your own computer at all, see OPTIONAL: NO SSH -- EVERYTHING IN THE BROWSER, after the Vast.ai section; the provider sections below assume ssh.

Make an ssh key for the rented machine (one key is fine for all three providers; the .pub file is what you give them):

ssh-keygen -t ed25519 -f ~/.ssh/stkv_rent -N ''

(In Windows PowerShell, write $HOME\.ssh\stkv_rent for ~/.ssh/stkv_rent, here and below, and use -N '""' if '' is refused.)

OPTIONAL: BUILD AT HOME (send the rented machine only the program)

Skip this section unless you want it; most people will. It keeps the source off the rented machine, and it needs a Linux computer with Docker, or Ubuntu 24.04 with CUDA 12.6 or later. If you do it, follow the IF YOU BUILT AT HOME paragraph in your provider's section.

From inside the unpacked release folder, EITHER

Then pack everything the rented machine needs, which is everything but src/, plus the vocabulary file from src/:

tar cf ../stkv-pod.tar --exclude=./src --exclude=./models \
    --exclude=./compare_work .
tar rf ../stkv-pod.tar ./src/models/vocab.txt
tar tf ../stkv-pod.tar | grep '^\./src/.'

The last line must print exactly one line: ./src/models/vocab.txt

RUNPOD (Secure Cloud)

Prices and RAM, Secure Cloud, on-demand, read from runpod.io/pricing in October 2026 (the console shows the current price as you choose):

RTX A5000   24 GB   $0.27/hr    25 GB RAM
RTX 3090    24 GB   $0.50/hr   125 GB RAM
L4          24 GB   $0.59/hr    50 GB RAM
RTX A6000   48 GB   $0.59/hr    50 GB RAM
RTX 4090    24 GB   $0.89/hr    41 GB RAM
L40S        48 GB   $1.09/hr    94 GB RAM
RTX 5090    32 GB   $1.19/hr    35 GB RAM
A100 PCIe   80 GB   $1.79/hr   117 GB RAM

The RTX 4090 is the one the commands below are written for; the RTX A5000 is the cheapest that qualifies. Pods are billed by the second. Storage: the container disk is $0.10 per GB per month while the Pod runs, so 150 GB is about 50 cents a day. A Pod has no fixed hourly minimum and no fee for data in or out.

Two ways to do the same thing follow: THE CONSOLE (point and click, easier the first time) and THE COMMANDS (the exact lines, for a repeat or for someone who prefers a terminal). Both create the same Pod. Whichever you use, the steps ON THE POD and DELETE THE POD at the end apply. A third way, with no terminal on your own computer, is RUNPOD IN THE BROWSER under OPTIONAL: NO SSH -- EVERYTHING IN THE BROWSER, below.

THE CONSOLE

  1. Make an account at runpod.io, open Billing (console.runpod.io/ user/billing) and add credit; $25 covers an afternoon on an RTX 4090 with room to spare. Accounts have a spend limit of $80 an hour; one Pod is nowhere near it.
  2. Settings -> SSH Public Keys (RunPod has also called this page Credentials): paste the contents of ~/.ssh/stkv_rent.pub from your computer. Only the .pub file; never the other one.
  3. Pods -> Deploy. At the top, choose SECURE CLOUD, not Community Cloud. Pick the card (for example RTX 4090); the price and the RAM are shown on its tile; the location filter, when offered, lets you keep to US data centers (their ids start with US-).

  4. Under the template, choose Edit Template (the label has also been "Edit Pod Template" and "Customize deployment") and set:

    Container Image:          nvidia/cuda:12.8.1-devel-ubuntu24.04
Container Disk:           150 GB
Volume Disk:              0 GB
Expose TCP Ports:         22
Environment Variables:    PUBLIC_KEY = (paste stkv_rent.pub)
Container Start Command:  bash -c followed by a space and the
                          whole quoted line below, exactly

    That image has the CUDA toolkit and no ssh server, so the start command installs one as the Pod starts (RunPod's own SSH guide gives the same recipe). It is one line, quotes included:

    'apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-server && mkdir -p /run/sshd /root/.ssh && echo "$PUBLIC_KEY" >> /root/.ssh/authorized_keys && chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys && exec /usr/sbin/sshd -D'

    If the deploy form has a CUDA version filter (Additional filters -> CUDA Versions), allow 12.8 and later.

  5. Deploy On-Demand (not Spot/Interruptible: an interruptible Pod can be taken away mid-run). BILLING STARTS HERE. The Pod shows as starting while it downloads the image and installs the ssh server; a few minutes.

  6. Open the Pod's Connect tab. Under "SSH over exposed TCP" (also shown as "Direct TCP Ports": an address and a port, mapped to :22) is a line of the form

    ssh root@203.0.113.10 -p 17445 -i ~/.ssh/id_ed25519

    Run it from your computer with your own key file in place of ~/.ssh/id_ed25519 (~/.ssh/stkv_rent), and add nvidia-smi at the end the first time:

    ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS nvidia-smi

    "Connection refused" means the ssh server is still installing; wait a minute and try again. When it works it prints the card, and the "Driver Version" on its top line must be 570 or later. Do not use the other line on the Connect tab, "SSH" through ssh.runpod.io: that path has no scp, so you could not copy files over it.

  7. Copy the release file over (from the folder you downloaded it to) and log in:

    scp -i ~/.ssh/stkv_rent -P PORT streamingkv-free-*.tar.gz \
    root@ADDRESS:/root/
ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS

    (scp takes a capital -P for the port; ssh a small -p.) Then go to ON THE POD.

THE COMMANDS

Run the "(your computer)" lines in ONE terminal: later steps use the names (RUNPOD_API_KEY, POD, IP, PORT) that earlier steps set.

  1. (RunPod's web console) Make an account, add credit, and create an API key with read and write access (under Settings; RunPod has named the page "API Keys" and "Credentials"). Do not save the key in a file or paste it into a chat. (your computer):

    read -rs -p 'RunPod API key: ' RUNPOD_API_KEY; echo
export RUNPOD_API_KEY

  2. (your computer) Write the request for the Pod: Secure Cloud, one RTX 4090, in the US, Ubuntu 24.04 with the CUDA 12.8 toolkit, 150 GB of disk, ssh on port 22. The image has no ssh server, so the Pod installs one when it starts. For another card, put its name as RunPod spells it (for example "NVIDIA GeForce RTX 3090" or "NVIDIA RTX A5000") in gpuTypeIds (and, if you build at home, build for it).

    cat > pod_request.json <<EOF
{"name": "stkv-review", "cloudType": "SECURE",
 "computeType": "GPU", "gpuCount": 1, "interruptible": false,
 "gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
 "countryCodes": ["US"],
 "imageName": "nvidia/cuda:12.8.1-devel-ubuntu24.04",
 "containerDiskInGb": 150, "volumeInGb": 0,
 "allowedCudaVersions": ["12.8", "12.9", "13.0"],
 "ports": ["22/tcp"], "supportPublicIp": true,
 "env": {"PUBLIC_KEY": "$(cat ~/.ssh/stkv_rent.pub)"},
 "dockerStartCmd": ["bash", "-c",
"apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y openssh-server && mkdir -p /run/sshd /root/.ssh && echo \"\$PUBLIC_KEY\" >> /root/.ssh/authorized_keys && chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys && exec /usr/sbin/sshd -D"]}
EOF
python3 -m json.tool pod_request.json > /dev/null \
    && echo request ok

    The last line must print "request ok". (The long quoted line is one piece and must stay on one line; the backslashes in it are part of it.)

  3. (your computer) Create the Pod. BILLING STARTS HERE.

    curl -sS -A stkv-review -X POST https://rest.runpod.io/v1/pods \
    -H "Authorization: Bearer $RUNPOD_API_KEY" \
    -H 'Content-Type: application/json' \
    --data @pod_request.json -o pod.json
POD=$(python3 -c "import json; \
    print(json.load(open('pod.json')).get('id', ''))")
echo "Pod: $POD"

    If "Pod:" is followed by nothing, read pod.json: usually no card of that type is free on Secure Cloud in the US just then; pick another card in step 2 (and build for it), or try later.

  4. (your computer) Wait for the Pod's ssh address. It takes a few minutes: the Pod downloads the image, then installs the ssh server.

    while :; do
  curl -sS -A stkv-review \
      -H "Authorization: Bearer $RUNPOD_API_KEY" \
      https://rest.runpod.io/v1/pods/$POD -o podnow.json
  A=$(python3 -c "import json; p=json.load(open('podnow.json')); \
      m=p.get('portMappings') or {}; \
      print(p.get('publicIp') or '', m.get('22') or '')")
  set -- $A; [ -n "$2" ] && break; sleep 15
done; IP=$1; PORT=$2; echo "ssh address: $IP port $PORT"
ssh -i ~/.ssh/stkv_rent -o StrictHostKeyChecking=accept-new \
    -p $PORT root@$IP nvidia-smi

    "Connection refused" means the ssh server is still installing: wait a minute and run the ssh line again. When it works, it prints the card; the "Driver Version" must be 570 or later.

  5. (your computer) Copy the release file to the Pod, then log in:

    scp -i ~/.ssh/stkv_rent -P $PORT streamingkv-free-*.tar.gz \
    root@$IP:/root/
ssh -i ~/.ssh/stkv_rent -p $PORT root@$IP

ON THE POD (over ssh, from either path)

You are root and there is no sudo: wherever a line elsewhere in the README starts with sudo, leave that word out. The driver comes from RunPod's host and the CUDA toolkit is in the image. Install the rest, unpack and build:

apt-get update && apt-get install -y cmake g++ wget python3 \
    python3-venv
mkdir -p /root/stkv && tar xzf /root/streamingkv-free-*.tar.gz \
    -C /root/stkv --strip-components=1
cd /root/stkv
bash build/build.sh --accept-license

Then, from /root/stkv: the model lines in the README's INSTALLING BY HAND (mkdir, wget, sha256sum: 22 GB, several minutes on a data-center line), its Python lines (python3 -m venv, pip install), then RUN 1 and RUN 2 (the six planted facts), and the comparison from HOW THIS DIFFERS:

bash build/compare.sh --yes

(--yes answers its questions before it downloads the test model and llama.cpp.) The interactive program runs as in USE IT.

While the card is working (a SECOND terminal on your computer), read the link between the card and the rest of the machine:

ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS nvidia-smi \
    --query-gpu=pcie.link.gen.current,pcie.link.width.current \
    --format=csv -l 1

Stop it with Ctrl-C. Read it while the card is busy: an idle link drops to its slowest setting, and the ".max" fields report what the card supports, not what its slot gives it. Rented machines often seat a card in a slower or narrower slot than it can use; we have seen an RTX 5090 in a PCIe gen 4 slot and an RTX 4090 on 8 lanes instead of 16. The part of the context that is not on the card crosses this link for every token written, so report the generation and the lane count with any speed you quote.

Keep what you want before the disk goes: compare.sh prints its results, so save that terminal's text. Its tables and logs, without the llama.cpp source and build in the same folder (thousands of files, slow to copy one by one), come back as one file (your computer):

ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS 'cd /root/stkv && \
    tar czf - compare_work/RESULT.tsv \
    compare_work/LONG_RESULT.tsv compare_work/qwen*' \
    > compare_results.tgz

DELETE THE POD

Console: Pods -> the Pod's menu -> Terminate. NOT Stop: a stopped Pod is still yours, its container disk is wiped but it stays in your list, and a volume disk (if you gave it one) bills at $0.20 per GB per month while stopped. Terminate deletes everything and ends the bill.

Commands (your computer):

curl -sS -A stkv-review -X DELETE \
    -H "Authorization: Bearer $RUNPOD_API_KEY" \
    https://rest.runpod.io/v1/pods/$POD
curl -sS -A stkv-review -o /dev/null -w '%{http_code}\n' \
    -H "Authorization: Bearer $RUNPOD_API_KEY" \
    https://rest.runpod.io/v1/pods/$POD

The second line must print 404. Either way, then look at the Pods page of RunPod's console: nothing of yours may be listed there. If something is, Terminate it there.

IF YOU BUILT AT HOME: copy ../stkv-pod.tar instead of the release file, and on the Pod unpack it with

mkdir -p /root/stkv && tar xf /root/stkv-pod.tar -C /root/stkv

in place of the tar xzf line, and leave out the build.sh line.

AMAZON WEB SERVICES (EC2)

An EC2 instance is a whole virtual computer that Amazon runs, so the README applies to it exactly as to your own Ubuntu machine: you log in as the user ubuntu, sudo works, and build/setup.sh can install the driver and everything else.

To reach it from a browser tab instead of ssh, see AWS IN THE BROWSER under OPTIONAL: NO SSH -- EVERYTHING IN THE BROWSER, below.

INSTANCE TYPES THAT QUALIFY. The letter g families carry the cards; within a family the size sets the vCPUs, the RAM and the price, and the card is the same. Read from the AWS instance-type pages and the EC2 On-Demand pricing page (Linux, US East, October 2026):

g6.2xlarge    L4     24 GB    8 vCPU    32 GiB RAM   about $0.98/hr
g5.2xlarge    A10G   24 GB    8 vCPU    32 GiB RAM   about $1.21/hr
g6e.xlarge    L40S   48 GB    4 vCPU    32 GiB RAM   about $1.86/hr
g6e.2xlarge   L40S   48 GB    8 vCPU    64 GiB RAM   about $2.24/hr

g6.2xlarge is the recommendation: the cheapest with 32 GiB of RAM. The .xlarge sizes of g5 and g6 have only 16 GiB of RAM (about $0.80/hr): avoid them. The .4xlarge and larger sizes give more RAM for more money and the same card. REFUSED: g4dn (T4), g4ad (AMD), p3 (V100), p2 and anything older; the program will not start on them.

Disk: the root volume is an EBS gp3 volume, about $0.08 per GB per month in US East, so 150 GiB is about $12 a month, 40 cents a day, and is billed until the volume is deleted, whether the instance runs or not.

  1. Make an account at aws.amazon.com (a card is required; AWS bills it monthly for what you used). Sign in to the console and, in the top bar, set the region to US East (N. Virginia) or US West (Oregon): both are in the US and both carry the g6 family. Everything below is per region, the quota included; stay in the one you picked.

  2. THE QUOTA. A new account may run ZERO vCPUs of GPU instances until it asks. Open the Service Quotas console (console.aws.amazon.com/ servicequotas), choose AWS services -> Amazon Elastic Compute Cloud (Amazon EC2), search for the quota named

    Running On-Demand G and VT instances

    (its code is L-DB2E81BA), and choose Request increase at account level. Enter 8 (the vCPUs of a g6.2xlarge; enter 16 if you may want a .4xlarge). Small requests are often approved within hours, automatically; others go to Support and can take a day or two, and the request is per region. Until it is approved, launching the instance fails with a message about vCPU limits. Request it first, then do steps 3 and 4 while you wait. (Spot instances have a separate quota, and a Spot instance can be taken away mid-run: use On-Demand.)

  3. A key pair. EC2 -> Network & Security -> Key pairs -> Create key pair: name stkv-aws, type ED25519, format .pem. The browser downloads stkv-aws.pem once; move it to ~/.ssh/ and

    chmod 400 ~/.ssh/stkv-aws.pem

    (ssh refuses a key file anyone else can read). Or, to use the key you made earlier (WHAT YOU NEED...): Actions -> Import key pair and paste ~/.ssh/stkv_rent.pub.

  4. Launch. EC2 -> Instances -> Launch instance:

    Name: stkv-review
Application and OS Images: Quick Start -> Ubuntu, and in the
    list pick the Ubuntu Server 24.04 LTS image (64-bit x86).
    Not 22.04, not 26.04 for a first run, not Ubuntu Pro.
Instance type: g6.2xlarge
Key pair (login): stkv-aws (or the one you imported)
Network settings: Create security group, Allow SSH traffic from
    -> My IP. Nothing else open.
Configure storage: 150 GiB, gp3.
Advanced details: leave alone.

    Launch instance. BILLING STARTS HERE. If it fails at once with a vCPU-limit message, the quota (step 2) is not yet approved. Within a minute or two the instance shows Running; its Public IPv4 address is on its row.

    WHY THE PLAIN UBUNTU IMAGE. This release is tested on Ubuntu 24.04 as Canonical publishes it, and build/setup.sh installs the driver and the toolkit it has checked. AWS also publishes a "Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04)" with the driver and several CUDA toolkits already installed; it saves the driver install and the reboot, but we have not run this release on it and its toolkit layout is its own, so it is a second choice if you know it already.

  5. Connect, from your computer (the first time, answer yes to the fingerprint question):

    ssh -i ~/.ssh/stkv-aws.pem ubuntu@ADDRESS nvidia-smi

    On a plain Ubuntu image this prints "command not found": there is no driver yet, which is expected; the ssh connection working is the check. A timeout means the security group does not allow your address (your IP changed, or you are on a different network): edit the security group's inbound rule to My IP again.

  6. Copy the release file over (from the folder you downloaded it to), log in, unpack it and run the setup:

    scp -i ~/.ssh/stkv-aws.pem streamingkv-free-*.tar.gz \
    ubuntu@ADDRESS:~/
ssh -i ~/.ssh/stkv-aws.pem ubuntu@ADDRESS
mkdir -p ~/stkv && tar xzf ~/streamingkv-free-*.tar.gz \
    -C ~/stkv --strip-components=1 && cd ~/stkv
bash build/setup.sh

    setup.sh checks the machine, installs what is missing (the driver, the CUDA toolkit, the Python packages), downloads the model, builds, and offers the six-question check, asking before each step. When it installs the driver it asks you to reboot (sudo reboot); wait a minute, ssh in again (the address does not change on a reboot), cd ~/stkv and run bash build/setup.sh again. Then

    bash build/compare.sh --yes

    for the comparison, and bash build/start.sh for everything else. Read the PCIe link as in the RunPod section (ON THE POD), with this ssh line in place of that one; AWS seats these cards in the slot the instance type gives them, and the figures are what to report with any speed. The source is on the instance's disk until step 8.

    IF YOU BUILT AT HOME: copy ../stkv-pod.tar instead, unpack it with

    mkdir -p ~/stkv && tar xf ~/stkv-pod.tar -C ~/stkv && cd ~/stkv

    and follow the README's INSTALLING BY HAND, with sudo as written there: the driver (sudo ubuntu-drivers install, then sudo reboot), the Native toolkit lines for Ubuntu 24.04 (compare.sh builds llama.cpp with it), the model lines and the Python lines; then RUN 1, RUN 2 and bash build/compare.sh --yes.

  7. Keep what you want: compare.sh prints its results, so save the terminal's text, and the results file comes back with

    scp -i ~/.ssh/stkv-aws.pem \
    ubuntu@ADDRESS:~/stkv/compare_work/RESULT.tsv .
  8. TERMINATE. EC2 -> Instances -> select it -> Instance state -> Terminate (delete) instance, and confirm. Not Stop: a stopped instance keeps its disk and bills it. The root volume is deleted with the instance by default (Delete on termination: Yes, shown on the instance's Storage tab; check it there before you terminate). Then EC2 -> Elastic Block Store -> Volumes must list nothing of yours (an "available" volume there is one still billing; delete it), and Instances must show the instance as terminated, then gone. If you made an Elastic IP, release it too.

VAST.AI (Secure Cloud offers only)

Vast.ai lists machines offered by many separate owners, from one card in a house to racks in a data center, and lets you filter. The ONLY offers that meet rule 2 are those under its "Secure Cloud" filter, which Vast.ai describes as equipment of registered businesses in professionally managed facilities that it has verified; they carry a blue datacenter label on the offer. Everything without that label is a host's own machine and is out, however cheap, however high its reliability score. If no Secure Cloud offer in the US has a qualifying card when you look, use RunPod or AWS instead. Do not turn the filter off.

Like a RunPod Pod, a Vast.ai instance is a container started from a Docker image: you are root, there is no sudo, the driver is the host's, and the steps ON THE POD in the RunPod section apply unchanged.

To reach it from a browser tab instead of ssh, see VAST.AI IN THE BROWSER under OPTIONAL: NO SSH -- EVERYTHING IN THE BROWSER, below.

Prices are set by each owner, per second, and vary with the day; in October 2026 a 24 GB card was commonly between about $0.30 and $0.70 an hour across the marketplace, and the data-center offers tend to cost more than the rest. Storage is a separate per-GB charge shown on the offer, and continues while an instance is stopped; so does a per-byte charge for data in and out. Vast.ai is prepaid: it stops instances when the credit is gone but keeps charging storage against the balance.

  1. Make an account at cloud.vast.ai, open Billing, Add Credits ($25 is plenty for an afternoon). Keys (cloud.vast.ai/manage-keys): paste the contents of ~/.ssh/stkv_rent.pub. The key is given to instances you create after this, not to ones already running.

  2. Search. On the search page, turn on the filter labelled

    Secure Cloud (Only Trusted Datacenters)

    and set the location filter to the United States. Set the GPU filter to the card you built for (RTX 4090, RTX A6000, L40S ...), one GPU, and the disk space slider to 150 GB (it is also the size you get, and it cannot be changed after the instance is made). Each offer card shows the card, the RAM, a reliability score, a region and a price; check for the blue datacenter label on each one before you go on, and skip any offer with 16 GB of RAM or less.

  3. The template, in the upper left of the search page: Change Template, and pick one whose image is plain Ubuntu with the CUDA toolkit, or enter the image yourself with

    Image: nvidia/cuda:12.8.1-devel-ubuntu24.04
Launch Mode: SSH

    (the SSH launch mode starts an ssh server in the container for you; the image itself has none). Not Jupyter, not Entrypoint.

  4. RENT on the offer you chose. BILLING STARTS HERE. On the Instances page the new instance goes from loading to running in a few minutes. Its ssh connection string is behind the key or terminal icon on the instance; it has the form

    ssh -p 50983 root@203.0.113.20 -L 8080:localhost:8080

    Use it from your computer without the -L part and with your key:

    ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS nvidia-smi

    Answer yes to the fingerprint question; the Driver Version must be 570 or later. Two kinds of connection are offered, Direct and Proxy; Direct is faster and is the one to use for the 22 GB download and the copies.

  5. Copy the release file over and log in:

    scp -i ~/.ssh/stkv_rent -P PORT streamingkv-free-*.tar.gz \
    root@ADDRESS:/root/
ssh -i ~/.ssh/stkv_rent -p PORT root@ADDRESS

    then follow ON THE POD in the RunPod section word for word (you land in a tmux session; that is fine). IF YOU BUILT AT HOME, follow the RunPod section's paragraph of that name, with this address and port.

  6. DESTROY the instance when done: on the Instances page, the trash icon (Destroy), not the square one (Stop). Stop keeps the instance and its data and keeps charging storage; Destroy deletes the instance and all its data and ends every charge for it. Then the Instances page must list nothing of yours.

OPTIONAL: NO SSH -- EVERYTHING IN THE BROWSER

Skip this section if ssh and scp are fine with you: they are the main path above, they work from Windows PowerShell as well as from Linux and macOS, and they are the better tool for a run that lasts hours. This section is for someone who would rather not open a terminal on their own computer at all. Everything the program does happens in a terminal (the menus, the chat, the comparison), so a terminal shown in a browser tab is all that is needed, and each provider has one. Two things change: how you reach the machine's terminal, and how the release file gets onto the machine without scp. After that, the steps ON THE POD (RunPod, Vast.ai) and AWS steps 6 to 8 are the same as above, typed into the browser tab. The labels and behaviour below are from each provider's documentation, read in October 2026; we have not yet run this path end to end ourselves, and where the documentation is silent this says so.

A BROWSER TAB IS NOT A TERMINAL YOU CAN WALK AWAY FROM. Closing or reloading the tab ends the shell inside it, and with it anything that was running there; RunPod's documentation says as much of its web terminal ("not recommended for long-running processes"). So run every long step (build.sh, the 22 GB model download, compare.sh) inside tmux, which keeps running when the tab goes. Before the first long step:

tmux new -s stkv

and type the commands as usual. If the tab closes, open a terminal again and

tmux attach -t stkv

brings the same session back, output and all. (Ubuntu's server image and RunPod's image both have tmux; if it says command not found, add tmux to the apt-get line of your section.)

THE RELEASE FILE WITHOUT SCP

The download link you were sent works once. The page it shows after you accept carries a second address, which is the file itself, and that is the address the rented machine can fetch.

  1. On your own computer, open the link, read, and accept, ONCE. Your browser downloads your copy; keep it. The page that then says "Accepted -- your download is starting" has a "click here" link in case the download did not start: right-click it and choose Copy link address (or Copy link). That address fetches the file itself, and keeps working for 24 hours after you accepted, a handful of times. Do not open the original link again; it was spent when you accepted.

  2. In the rented machine's browser terminal, in the folder the steps expect (/root on RunPod and Vast.ai; your home folder on AWS):

    wget --content-disposition 'PASTE THE COPIED ADDRESS HERE'

    with the address inside the quotes. It saves the file under its own name, streamingkv-free-...tar.gz, so the tar lines above apply unchanged. The download service answers only to addresses in the United States, which rule 4 already asks of the machine; from anywhere else it refuses. If wget is "command not found", run the apt-get line from ON THE POD first.

  3. If the 24 hours have passed, or the address was lost: on RunPod and Vast.ai the JupyterLab file browser (below) has an upload button that takes the copy from your own Downloads folder; the file is about 1.3 MB, so no size limit comes into it. On AWS the browser terminal has no upload button (AWS CloudShell has one, but it is a separate small machine, not your instance): ask for a new link, or use the ssh path.

RUNPOD IN THE BROWSER

docs/RUNPOD.md is this path step by step, walked on a real machine and kept current: follow it rather than what is below.

RunPod's Connect tab has a web terminal and, on templates that include it, JupyterLab (a file browser with an upload button, plus terminals). RunPod's documentation does not say which images carry the web terminal; RunPod users report that its Start button does nothing on an image without an ssh server, and the plain nvidia/cuda image used above has none. So this path uses RunPod's own image instead. It is built on the same nvidia/cuda 12.8.1 Ubuntu 24.04 image and adds the ssh server, JupyterLab, cmake, g++, wget and tmux (read from its Dockerfile):

runpod/base:1.4.1-cuda1281-ubuntu2404

That tag was published the week this was written; the Tags page at hub.docker.com/r/runpod/base lists the current ones. Take the newest whose name ends in cuda1281-ubuntu2404.

  1. Follow THE CONSOLE steps 1, 3 and 5 of the RunPod section; step 2, the ssh key, is not needed. In step 4 set instead:

    Container Image:          runpod/base:1.4.1-cuda1281-ubuntu2404
Container Disk:           150 GB
Volume Disk:              0 GB
Expose HTTP Ports:        8888
Expose TCP Ports:         22
Environment Variables:    JUPYTER_PASSWORD = (a password you
                          choose, 16 characters or more)
Container Start Command:  leave empty (the image has its own)

    If the deploy form shows a checkbox "Start Jupyter notebook", leave it ticked; it is RunPod's way of setting the same thing. RunPod's notes on the image say JupyterLab does not start without JUPYTER_PASSWORD, and that port 8888 is reachable by anyone who has its address and the password, which is why the password must be a real one. Never set JUPYTER_DISABLE_AUTH.

  2. Open the Pod's Connect tab. Under HTTP Services, "Jupyter Lab" (port 8888) opens JupyterLab in a new tab; enter the password. The file browser is the left panel; its upload button (an arrow pointing up, above the file list) takes streamingkv-free-*.tar.gz from your own computer, if you are not fetching it with wget. A terminal is File -> New -> Terminal, or the Terminal tile of the Launcher tab. It opens in JupyterLab's top folder, which is also where an upload lands; move the file to where ON THE POD expects it:

    mv streamingkv-free-*.tar.gz /root/

    The Connect tab's own web terminal (Start, then Open Web Terminal) is the other terminal; either does, and both end when their tab closes, hence tmux.

  3. Start tmux (above), then ON THE POD, word for word. For the PCIe reading, open a second terminal the same way instead of a second ssh. The results are text: in the terminal

    cat compare_work/RESULT.tsv

    and copy it from the screen, or right-click the file in JupyterLab's file browser and choose Download. Then DELETE THE POD as above.

AWS IN THE BROWSER

EC2 Instance Connect is a terminal in the EC2 console's own page. AWS's documentation says it is preinstalled on Ubuntu 20.04 and later images, so the Ubuntu Server 24.04 image above has it; no key pair is needed, and the account owner's sign-in already has the permission (an IAM user with limited rights needs ec2-instance-connect:SendSSHPublicKey and ec2:DescribeInstances). What it does need is a public IPv4 address, which the launch gives it, and a security group rule that lets the EC2 Instance Connect SERVICE in on port 22: the connection comes from AWS, not from your own address, so "My IP" alone is not enough.

  1. Launch as in AWS step 4 (keep the key pair if you may want ssh as a fallback). Then add the rule: EC2 -> Instances -> select the instance -> Security tab -> click its security group -> Edit inbound rules -> Add rule: Type SSH; Source Custom, and in the field next to it type ec2-instance-connect and pick

    com.amazonaws.us-east-1.ec2-instance-connect

    (us-west-2 in place of us-east-1 for Oregon; the list is per region) -> Save rules. The My IP rule from the launch can stay.

  2. Instances -> select the instance -> Connect -> the EC2 Instance Connect tab -> Connect using a Public IP; check that Username reads ubuntu -> Connect. A terminal opens in the browser tab, logged in as ubuntu. Your browser must be able to reach the address prod.<region>.oneclickv2-proxy.ec2.aws.dev on port 443; AWS notes that a firewall or VPN which blocks it fails the connection.

  3. Fetch the release file with the wget line above (in your home folder), start tmux, then AWS step 6 from its tar line onwards. setup.sh's reboot ends the browser session: wait a minute and press Connect again. For the PCIe reading, open a second Connect tab. Results:

    cat ~/stkv/compare_work/RESULT.tsv

    and copy the text from the screen. Then step 8.

VAST.AI IN THE BROWSER

Vast.ai's Jupyter launch mode puts JupyterLab on the instance, and the Open button on the instance card opens it in a browser tab; the file browser's upload button and File -> New -> Terminal are then as on RunPod. Vast.ai's documentation says this mode replaces the image's entrypoint with Vast.ai's own setup and "requires that your docker image is compatible with typical jupyter setup"; it does not say whether the plain nvidia/cuda image qualifies, and we have not tried it. So: in step 3 of the Vast.ai section choose Launch Mode Jupyter instead of SSH, leave "Jupyter direct HTTPS" off (the default is a proxy; the direct option asks you to install Vast.ai's certificate, and the only file going up is 1.3 MB), and rent. If the instance card reaches Open within a few minutes, click it, open a terminal, move the uploaded file to /root/ as on RunPod, start tmux and follow ON THE POD. If it sits at loading well past that, Destroy it (it bills meanwhile) and either use RunPod's browser path, whose image is documented to carry JupyterLab, or the ssh path above.

WHEN SOMETHING DOES NOT MATCH

The screens of all three providers change; the commands change much less. If a label here is not on your screen, look for the nearest one that means the same (Terminate / Delete / Destroy; Secure Cloud / Trusted Datacenters; Expose TCP Ports / TCP ports). If a price here is far from what you see, trust the screen. If a card you want is not offered on a qualifying tier on the day, pick another card from the list above and build for it, or try another provider, rather than taking a tier that rule 2 excludes. Report problems with the free edition itself as CONTRIBUTING says, with the card, the provider, the RAM and the PCIe link figures.

StreamingKV LLC, Dike, Texas. Home. Privacy notice.