transition2.aiStart a project

Models, Tooling & Developer Practice

The Model Ladder for Coding Agents

Start coding agents on a small model, climb one rung only after two real failures, and let a reviewer one rung higher check the work.

Eric MacDougall ·

Answer first

A model ladder is an ordered set of model-and-effort settings with explicit rules for moving up. Start every coding task on the smallest rung. Move up one rung only after two genuine failures at the current rung, where "genuine" means a machine receipt shows a review that blocked with named findings or tests that still fail after a claimed fix. Let a reviewer one rung higher check the work, preferably from a different provider. Make the top rung require a recorded human decision. This is a different question from model routing for cost: routing picks a model per request, while the ladder decides when a stuck task may spend more.

The rungs

In the open-source agent harness the table has seven rungs, R0 to R6. In general terms:

  • R0: a small model, the default for every task.
  • R1 to R3: a larger model at medium, high and extra-high effort. All writable.
  • R4: the larger model at maximum effort. Review only.
  • R5: the top model at medium effort. Writable, but needs the user's permission.
  • R6: the top model at high effort. Review only.

Review-only rungs exist so that a strong model can check work without being the one that writes it. The climb skips them. The table also sets two numbers: two failures per rung, and at most three climbs before something else must happen.

What counts as a failure

The most important rule is that failures come from machine receipts, not from anyone's description of events. The ledger counts:

  • A review that blocked with named findings. A block with no named finding is not counted.
  • Tests that failed after a fix was claimed.

It does not count transient infrastructure errors, low disk, quota or sign-in failures, timeouts, or anything it cannot classify. Replaying the same receipt twice is a no-op, because receipts are keyed by a hash. The same finding at the same commit with no fix claimed in between counts once. A new, different finding restarts the count, because a task that fails for a new reason is making progress, not stuck.

Who reviews whom

The reviewer sits one rung above the author, and the other harness is preferred: a reviewer from a different provider is less likely to share the author's blind spots. If that harness is unavailable, the same harness at the reviewer rung is used. If the author's rung is unknown, treat the author as the lowest writing rung and reset to it, recording why. A review-only rung needs no reviewer. For how reviews are bound to a stage and a commit, see Review Gates for AI-Written Code.

Enforcement and spend

A ladder in a document is advice. Per the source, and in a separate lab that we do not reproduce here, the harness enforces it at the moment an agent is spawned: with no model named, the task's rung is applied; a request above the rung is downgraded; a request below it is denied unless a reason is given; a review-only rung is denied for writing; and the top rung is denied unless permission is recorded. The shipped mode is warn, which reports "would deny" without blocking, so a team can see the rules working before turning enforcement on.

A separate, opt-in spend guard handles repeated expensive runs. It refuses an identical invocation unless one justified repeat is requested, requires a gate or a stated reason for a full-suite run, and, per the source (not run in our labs), caps attempts that launch nothing. Its receipts are local evidence, not an attestation and not global enforcement.

What we ran

From our lesson labs: Python 3.12.3, standard library only, in a throwaway home directory with no model launched. The toy ladder and its tests, then planted bugs that must each break the suite:

Ran 11 tests in <t>s

OK
mutant INFRA: FAILED (failures=1)
mutant NAMED: FAILED (failures=2)
mutant RESTART: FAILED (failures=1)
mutant HEAD: FAILED (failures=1)
mutant THRESHOLD: FAILED (failures=6)
mutant REPLAY: FAILED (failures=1)
mutant REVIEWER: FAILED (failures=1)
mutant PERMISSION: FAILED (failures=1)

Then the real table and the real ledger at commit 504d930, fed with fixture receipts we wrote ourselves (they are shaped like spec-review records and are not real reviews):

R0 effort=medium writable=True  permission=False reviewer=R1
R1 effort=medium writable=True  permission=False reviewer=R2
R2 effort=high   writable=True  permission=False reviewer=R3
R3 effort=xhigh  writable=True  permission=False reviewer=R4
R4 effort=max    writable=False permission=False reviewer=None
R5 effort=medium writable=True  permission=True  reviewer=R6
R6 effort=high   writable=False permission=True  reviewer=None
failures_per_rung = 2  max_rung_climbs = 3
first block, finding F1:
rung R0 | failures at rung 1 | climbs 0 | permission needed False | blocked_by [] | FAIL spec review blocked with findings
a different finding F2 (the count restarts, no climb):
rung R0 | failures at rung 1 | climbs 0 | permission needed False | blocked_by [] | FAIL spec review blocked with findings
F2 again (second failure of the same finding):
rung R1 | failures at rung 0 | climbs 1 | permission needed False | blocked_by [] | FAIL spec review blocked with findings
the same receipt replayed:
rung R1 | failures at rung 0 | climbs 1 | permission needed False | blocked_by [] |   (replayed)
block with no named finding:
rung R1 | failures at rung 0 | climbs 1 | permission needed False | blocked_by [] | NOT_COUNTED spec review without a usable verdict

Further blocks climbed through R2 and R3, and the next climb landed on R5 with permission needed True and blocked_by ['max_rung_climbs'].

Not run: the live spawn hook, the top-rung approval path, and the rescue launcher. The outage filter described above is documented for review and test receipts and was not exercised by the fixtures.

Decision framing

A ladder costs setup and discipline. It pays when agents run unattended long enough that "just use the biggest model" becomes the default and the bill, not the results, is what grows. It does not make a small model smarter; it makes the decision to spend more explicit and evidence-based. For small, fast tasks the simplest model is usually enough, see Choose the Simplest Model.

Risks and limits

  • The ladder assumes careless or hurried agents, not hostile ones. An agent that can edit the installed hooks can defeat the gates.
  • Fixture receipts show the arithmetic of the ledger, not how often real tasks fail.
  • Two failures per rung is a policy choice, not a measured optimum. Tune it against your own receipts.
  • A different provider as reviewer is a preference, not a guarantee of independence.

FAQ

What is a model ladder?
A model ladder is an ordered list of model and effort settings, from a small, cheap one to the strongest one you allow, together with rules for when a task may move up. Each rung is either writable, meaning an agent may write code with it, or review-only.
When should a coding agent escalate to a bigger model?
After the same material finding blocks review twice, or tests still fail after a claimed fix, at the current rung. One failure is not enough, and a failure caused by an outage, a quota limit or a timeout does not count.
Who should review the work of a coding agent?
A reviewer one rung above the author, from a different harness or provider when one is available. If the author is unknown, treat it as the lowest writing rung. A review-only rung needs no further reviewer.
Why should the top model need human permission?
It is the most expensive and least reversible spend. Requiring a recorded user decision before it is used turns a quiet cost escalation into an explicit choice.

Sources