Two Agents Are Better Than One
Past the hype, AI models still have a long way to go. They are genuinely great at synthesizing context and pulling a document together. They are not always as good as advertised, and you notice this most once the euphoria of the first few weeks wears off.
I have been using AI for many years and LLMs constantly since 2021. The thing that has annoyed me most is not the hard failures. It is how many simple mistakes they make that a little more attention would have caught. An off-by-one in a boundary condition. A regex that quietly accepts a trailing newline. A test that asserts the same wrong thing the code does. When I break a task into smaller pieces, or point an agent at one specific concern, it finds these things without difficulty. Left to run end to end on its own, it does not.
The idea
A few months ago I started using two agents for my development work instead of one. The first implements the tool and writes the tests. A second, running separately, reviews what the first one did and looks for exactly those small avoidable bugs, before I do my own high-level review by hand.
Over time my impression was that this caught a lot of the annoying stuff and made working with agents considerably more pleasant. But an impression is not evidence, and I wanted to be sure I was not hallucinating the benefit myself. So I benchmarked it.
The benchmark
I took 20 Python tasks from BigCodeBench and ran each one two ways.
- agent-pair. An implementer writes
solution.pyand its tests from the task description. A reviewer, in a separate session with no memory of the first, raises findings with evidence. The implementer applies what it accepts and rejects the rest with a reason. Anything it rejects goes to an orchestrator that decides who was right, running code first when running something would settle the argument. If the orchestrator rules against the implementer, the implementer is sent back once to carry out that ruling. One pass, no looping. - baseline. One agent writes the code and its tests and declares itself done.
Both arms get only the task description. Every role in a run uses the same model unless I say otherwise, so the difference between the arms is the review mechanism rather than one side getting a better model. Afterwards, both solutions are scored against BigCodeBench's own tests, which neither agent ever sees. 102 tests across the 20 tasks. Every task's reference solution passes all of its own tests, so nothing below is the benchmark's fault.
Results
| implementer | reviewer | agent-pair | baseline | gap | pair cost | baseline cost | pair time | baseline time |
|---|---|---|---|---|---|---|---|---|
| Sonnet | Opus | 78.4% | 70.6% | +7.8 | $10.38 | $2.39 | 44.8 min | 10.7 min |
| Opus | Opus | 91.2% | 79.4% | +11.8 | $15.15 | $4.62 | 51.4 min | 16.3 min |
Percentages are held-out tests passed. Costs and times are totals for all 20 tasks run one after another, not per task. Models: Opus is Claude Opus 5, Sonnet is Claude Sonnet 5.
The reviewer was not short of material. Across 20 tasks it raised 42 findings reviewing Opus's code and 62 reviewing Sonnet's, and the implementer accepted almost all of them. Genuine disagreement was rare: across these 40 runs the implementer pushed back 5 times, and the orchestrator ruled against it once.
But the cost
At first glance the overhead looks bad, and it is real: roughly three to four times the money and three times the wall clock. That is the honest headline number and I am not going to dress it up.
But the comparison is one baseline attempt against one reviewed attempt, and nobody ships the first draft of a failing task. To get the baseline to where the pair arm already is, you run it again on what it got wrong, and again, and each of those attempts costs roughly what the first one did. Add the time you spend reading the output yourself to find the boundary condition it missed. Once you count those, I think the gap closes a long way, and for my own work it is clearly worth it.
One result that cuts against me, which I will state plainly: Sonnet with an Opus reviewer reaches 78.4% for $10.38, while Opus writing alone reaches 79.4% for $4.62. On this benchmark, buying an expensive reviewer for cheap code costs more than simply using the expensive model to write it in the first place. The configuration that actually wins is Opus reviewing Opus.
The tool
The harness, the benchmark tasks and the scoring are all on GitHub at github.com/rehsani/agentpair. Every run records what each role said, what it accepted or rejected and why, how disputes were settled, and the tokens, cost and time behind all of it.
Take a look, and let me know if you have suggestions for making it better or cheaper. The obvious target is that the implementer currently starts a fresh session for each of its turns and re-reads files it wrote minutes earlier, which is a large part of the premium and is not fundamental to the idea.