Halv used 51% fewer tokens per correct answer on SWE-rebench

Same model. Same tasks. Same verifier. More correct answers from less budget.

Halv did more than make coding-agent runs smaller. It converted tokens into correct answers more efficiently.

Across the first 20 paired SWE-rebench tasks, the Halv arm:

• used 30.2% fewer tokens overall;
• cost 24.1% less overall;
• solved 10 tasks, compared with 7 for vanilla;
• used 51.1% fewer tokens per correct answer; and
• cost 46.9% less per correct answer.

The last two numbers matter most. A cheap wrong answer is still wrong. A short run that fails its verifier does not complete the work.

Tokens per correct answer measure how efficiently each system converts a finite budget into verified results.

The result that matters

Vanilla spent more than nine million tokens per correct answer. Halv spent 4.4 million.

Halv produced 43% more correct answers while using less total budget. That equals about 2.05× more correct answers per token.

What we tested

SWE-rebench pairs repository change requests with automated repository verifiers. We ran two arms against the same 20 tasks:

• Vanilla: Codex without the Halv toolkit.
• Halv: Codex with the Halv toolkit enabled.

Both arms used gpt-5.6-luna, medium reasoning, priority service, and Codex 0.152.0. Each pair used the same task verifier.

The Halv arm added three components:

• Crux supplied a prebuilt SCIP code map when the repository passed its language and size gates.
• RTK 0.42.4 reduced command output before it entered the model context.
• Halv Engine Headroom processed model requests and recorded token receipts.

Halv did not reuse cached model responses during this benchmark. We recorded SHA-256 fingerprints for the exact Crux and RTK executables in every run.

The Crux build derives from version 0.9.0 and includes a reproducible compatibility patch.

This benchmark measures the complete Halv toolkit. It does not assign the result to one component in isolation.

Why only 20 tasks?

This article reports the first 20 complete task pairs in the planned benchmark sequence. We chose this checkpoint because repository builds and verifiers take time.

Elapsed benchmark time defined the cutoff. We did not rank tasks by reward or token savings before selecting the checkpoint.

Future results will belong in a later report. This article will not silently expand this sample.

All 20 paired results

A reward of 1 means the repository verifier accepted the answer. Positive token savings mean Halv used fewer tokens.

Aggregate costs use unrounded run values. Task costs below are rounded to five decimal places.

Halv used fewer tokens on 12 of 20 tasks. It produced three correct answers that vanilla missed.

Vanilla did not solve any task that Halv missed in this sample.

Halv was not smaller on every task. It used more tokens on eight tasks. Two larger runs produced correct answers that vanilla missed.

The result is an aggregate efficiency gain. It does not promise that every Halv run will use fewer tokens.

Why might Halv be more correct?

This benchmark measures outcomes. It does not prove one causal mechanism. However, the three Halv-only wins support a useful working theory.

Halv changes the model’s information environment:

1. Crux lowers search uncertainty. Semantic references and outlines expose the likely change boundary before broad text search fills the context.
2. RTK removes repeated output. The model sees less build noise and retains more relevant evidence during long debugging loops.
3. Headroom protects useful context. The task, architecture, and test failures remain salient when a run requires several iterations.

All three Halv-only wins had a valid prebuilt Crux index. Each run queried that index.

In ArcadeDB-4455, both arms found the transaction-lock problem. Halv completed the transaction-scoped fix and passed the concurrency verifier.

The vanilla run required more shell calls and left one verifier failure.

In fallow-821, Halv followed the repository’s plugin architecture. It covered dependency activation, imported helpers, and workspace boundaries.

Vanilla added a broader filename exception but missed two required behaviors.

In fallow-824, Halv traced the mismatch between the original package name and its resolved pnpm store path.

It fixed the central resolver and added focused resolver tests. Vanilla did not cover the low-level cases.

This evidence fits a simple theory. Better navigation helps the model edit the correct abstraction. Cleaner context helps it preserve the full constraint set.

The theory is not that Halv always thinks less. Halv used more tokens on two of its three unique wins.

Those runs converted extra work into correct answers.

Model output remains stochastic, and three differential wins form a small sample. A causal study needs several seeds and component-level ablations.

Why correctness changes the cost story

Total spend understates the result. Halv reduced recorded cost by 24.1%, but it also produced three additional correct answers.

When we divide spend by correct answers, the cost reduction becomes 46.9%. The same pattern appears in tokens:

Vanilla: 63,438,109 tokens /  7 correct = 9,062,587 tokens per correct answer
Halv:    44,289,991 tokens / 10 correct = 4,428,999 tokens per correct answer

This is the practical benchmark question: how much budget does each system consume for an answer that survives its verifier?

Limits

Twenty task pairs can show an early signal. They cannot predict the same percentages for every repository, model, or task distribution.

This sample used one model configuration. Model output is stochastic. A new run can produce a different patch, token count, or reward.

All 40 selected records share the same model, effort, service tier, Codex version, dataset source, plan fingerprint, and immutable task references.

Treat these numbers as measured results for this sample, not as a universal guarantee.

Proof: audit each result independently

Each SWE-rebench task is an independent evaluation unit. The evidence preserves that boundary, so the proof unit is one task pair.

The planned sequence completed across multiple sessions because repository builds and verifiers take time.

Each table row maps one completed vanilla record to one completed Halv record. You can authenticate one pair without trusting the other 19 pairs.

The public evidence bundle includes these files for each selected arm:

1. The SWE-rebench task name, immutable reference, and task checksum.
2. The exact model, reasoning effort, service tier, Codex version, and arm configuration.
3. The Harbor verifier result that produced the reward.
4. The Codex event log with input, cached-input, and output token totals.
5. The Halv proxy receipt with Halv token accounting.
6. The Crux gate result, index metadata, cache status, and pinned binary hash.
7. SHA-256 manifests for every published evidence file and the complete archive.

The bundle also contains a small verifier that rebuilds the table from the 40 selected records.

Published copies redact local paths and secret-shaped strings. The process does not change metrics, rewards, tool calls, verifier output, or model output.

tar -xzf halv-swe-rebench-20-pairs-evidence.tar.gz
node verify-selected-pairs.mjs

The evidence authenticates the task, configuration, token count, recorded cost, and verifier reward for each row.

Authentication is not exact reproduction. A new stochastic model run can legitimately produce a different result.

Evidence bundle: Download the complete archive

Selected-pair manifest: Inspect selected-pairs.json

Browse evidence: Open all 20 task pairs

SHA-256 checksums: Inspect MANIFEST.sha256

Takeaway

Halv used less total budget and solved more tasks. The headline is not only 30.2% fewer tokens.

It is 51.1% fewer tokens per correct answer and 46.9% lower cost per correct answer.

That is the optimization we want: more verified work from the same budget.