Crux in real sessions: 90% correct, 47% cheaper per answer

In real sessions, the code map answers 90% of questions correctly and cuts the cost of each correct answer nearly in half.

Last time we benchmarked Crux — our code-index MCP server — against plain grep, one question per fresh session. Crux won on accuracy (96% vs 66%) and on cost per correct answer (−24%). And we ended with a promise: those numbers were the floor, because every question paid the full setup cost alone. Real coding sessions ask many questions about the same code, and the map only has to load once.

This is that measurement. Six questions per continuous session, four sessions per repo, on django (344k lines) and sympy (770k lines). Same model, same token meter, same scoring.

The results

Across all 48 questions, the agent with Crux answered 43 of 48 correctly (90%). With grep alone: 28 of 48 (58%). And each correct answer cost 31,644 tokens against grep’s 59,394 — 47% less.

The per-repo split tells you where the map earns it:

Read the sympy line again. It is twice django’s size, the repo where grep gets every other answer wrong — and the map’s advantage grows: 59% less spent per correct answer. The harder the codebase, the more the map is worth.

One session deserves its own sentence: on sympy, the agent went six for six — a perfect session on a 770k-line repo — for 169,693 tokens. A grep session on the same repo averages 248k tokens to get four answers wrong.

Why sessions change the economics

In the single-question benchmark, every answer carried the full cost of loading the map. In a session, that cost is paid once and every question after it rides free. In matched sessions, the per-question premium over grep melts as the conversation grows: +29% on question one, +7% by question three, +0.3% by question six. From the third question on, you are getting index-grade answers at grep prices.

And the flat profile is new. Every Crux session — on both repos — settled at 13–14 turns with exactly 4 index calls. No retry loops, no drill-downs, no re-fetching. The agent asks, the map answers, the session moves on.

The part we fixed to get here

Honesty requires one paragraph of history: the first time we ran sympy sessions, Crux cost 2.3× grep. The answers were right, but the map talked too much — uncapped candidate lists, a source snippet on every reference, and truncation text that told the agent to re-request everything it had already seen. Crux 0.6.2 rebuilt how results are shaped: ranked and capped ambiguity with auto-resolve, grouped references, real pagination, relevance ordering. Same index, same answers — a third of the tokens. The technical companion has the full before/after and every session’s numbers.

Where this lands

A wrong answer from a coding agent is a token bill deferred — the retry, or the refactor built on a caller list that missed three callers. In real sessions, on real codebases, the map now delivers the right answer and the smaller bill, and the advantage compounds with codebase size.

Crux is one layer of Halv and ships built in, activating automatically on projects big enough to need it. It is also open source, MIT, fully local — Crux 0.6.2 is out today: github.com/pedr0v/crux.