rtk on and off, same work, and my bill went up

rtk is a Rust tool that sits between a coding agent and the shell. It trims command output before the agent reads it, and its own counter, rtk gain, tells you how many tokens it saved. On 6 November 2026 I ran the same work with rtk on and with rtk off and compared what the runs cost. With rtk on they cost more.

How I ran it

• Same machine, same settings, hooks, rules files and model in both arms. 10 headless claude -p runs per arm, in ABBA order, all read-only work in one of my own repos.
• For the "off" arm I put a stand-in rtk first on PATH. Its hook call returned nothing, so every command ran unchanged, and an explicit rtk x ran x raw. The stand-in logged 65 hook calls and no direct calls, so rtk really was off.
• My runner had a bug. Every run did all five tasks in the task file: run the tests, read git history, summarise uncommitted work, find call sites, list file sizes. Both arms got the same bundle, so the comparison still holds.

What the runs cost

The dollar cost recorded for each run, 6 November 2026:

• Total, 10 runs: rtk on $4.20; rtk off $2.94
• Median run: rtk on $0.257; rtk off $0.215
• Mean without the one $1.36 outlier (an rtk-on run, 18 steps): rtk on $0.315; rtk off $0.294
• Median steps: rtk on 11; rtk off 11

The answers matched. Both arms found the same 5 failing tests and the same call sites. rtk came out cheaper on none of these measures. It cost 7% to 43% more, depending on how you treat the outlier. Ten runs a side is a hint, not proof.

It also gave wrong output

In an interactive session the same day, with rtk on, wc -l < file returned 0 for a file of 65 lines. grep -v '^hook' listed the matching lines when it should have listed the others. Run through rtk proxy, both commands gave the right answer. When a tool returns a wrong count, an agent given a wrong count may spend extra steps checking it; I did not measure that.

Two larger tests found a similar pattern

I read these posts. I did not re-run their tests.

• JetBrains, July 2026 (post). Claude Code with Sonnet 5 on SkillsBench, 86 tasks, 425 billed trials. The input that rtk compresses rose 3.2%, a change too small to call. In the low-effort run rtk reported 96 million tokens "saved" while the bill for the same trials went up. Their reasons: Read and Grep never go through rtk's Bash hook, and rtk counts savings as characters divided by 4, while most of the input is cheap cached re-reads.
• Quesma (post). Terminal-Bench, 1,740 attempts. 7% less input per turn, 18% more turns.

What I did with it

I removed the rtk hook. The number rtk gain prints is rtk's own estimate, so it can't show you a saving on your bill. If you want to know, run your own work with and without it and compare what you paid.

What this test does not cover: one machine, one repo, read-only tasks, one model setup. I did not test write-heavy work or other coding agents.