transition2.aiStart a project

Software Factories & AI-Assisted Engineering

Review Gates for AI-Written Code

Three review gates ask three different questions, and each approval only counts for one stage, one commit and one exact packet.

Eric MacDougall ·

Answer first

Use three separate review gates for AI-written code: a plan review before the code exists, a code inspection once the change and its focused tests exist, and a PR review of the frozen commit with full validation evidence. Only the last one counts as sign-off. Make every approval specific by requiring the review report to name its stage, the full commit it examined and a SHA-256 hash of the exact packet it read, and treat any report where one of those does not match the current state as no approval at all.

Why one "review" step is not enough

A single approval at the end of a pull request hides three different questions:

  1. Plan: is this the right thing to build, and will the checks prove it? Cheap to fix now, expensive later.
  2. Code inspection: does the change do what the plan said, and do the focused tests fail without the fix? Full validation can still be pending.
  3. PR: with the code, config and docs frozen, are the complete test results acceptable to merge?

When these collapse into one step, an early "looks fine" gets reused as a final approval, or a model that reviewed a plan is credited with having reviewed the code. Agents make this worse because they push often and their summaries sound confident. For how this fits a wider pipeline, see Building an Agent PR Review Pipeline You Can Trust and Human-in-the-Loop Code Review for Agent Output.

What makes an approval count

In the open-source agent harness we use for our own work, a review report is accepted only when a consumer checks all of these:

| Check | What it prevents | | --- | --- | | The review command exited 0 | Treating a refused, timed-out or invalid review as a pass | | Stage: matches the stage you asked for | A plan approval standing in for a PR approval | | Reviewed HEAD: equals the full current commit | Approval inherited by code pushed after the review | | Reviewed packet SHA-256: equals the hash of your packet file | Approving a packet that was edited after the reviewer read it | | The verdict marker belongs to that stage | A pass-looking line in the wrong report type |

The verdict markers differ per stage on purpose: plan reports use PLAN_RESULT, inspection reports use CODE_INSPECTION_RESULT, both with ACCEPT or BLOCK, and only the PR stage uses VERDICT: PASS. If an inspection report happens to quote the PR pass marker in its body, the helper replaces it with [PR signoff marker omitted]. That rewrite matches that exact string only, so it stops a quoted copy of the marker from passing as a verdict but is not a general defence, and a person still reads the report.

The hash is computed over the raw bytes of the packet, so even a trailing space changes it. A packet is also read once and verified again at the end of the review; a changed byte, or a bare touch of the file, makes the helper report that there is no current signoff. The packet check and the report are not atomic, and there is a known race window between them, which is one more reason a person still reads the result.

The packet is evidence, not a summary

The packet handed to a reviewer should be compact: scope, base and head, changed paths, finding ids, and the exact commands run with their exit status and counts. The helper runs a format lint that requires at least one command line with its exit status and a positive count. For example, python3 -m unittest discover followed by Ran 78 tests ... OK and exit 0. A sentence such as "tests pass" does not satisfy it. The lint is only a format guard: it cannot prove the run happened, so a person still has to read the evidence.

Size limits keep packets reviewable. Per the source (we did not run these limits here), the harness refuses a packet over 150,000 bytes, warns above 32 KiB, and lets the caller declare a tighter budget that is enforced before any model is launched. Put long material in a snapshot file and cite its path rather than pasting full files next to their diff.

When a reviewer fails

Review infrastructure fails in different ways and they should not all be handled the same way:

  • A quota or sign-in failure, or an unavailable model, may use at most one fallback reviewer from a different provider. (We ran the failure classifier on recorded error records, not a live provider.)
  • A rate limit, a spent budget, an invalid report or a plain unrecognized error does not trigger a paid fallback.
  • A material finding is a code problem, not a reviewer failure. Fix the code and review again; do not shop for a friendlier reviewer.
  • Each attempt gets a fresh output path. Per the source, the harness allows 720 seconds per provider and 1,440 seconds for the whole invocation; we did not run those deadlines.

Per the harness documentation (not run in our labs), the exit codes are the contract: 0 for an acceptance, 3 for a material block, 2 for no valid review at all. Keep the three apart in your automation.

The order that works

  1. Write the plan, name the checks that would prove it, and get the plan review.
  2. Implement. For each changed behavior, run a check that fails first against the old code or a deliberately broken variant.
  3. Run the focused checks, then ask for a code inspection, saying plainly that full validation is still pending.
  4. Repair named findings and re-run only the affected checks.
  5. Freeze code, config, fixtures and docs. Run the full validation once.
  6. Request the PR review on the exact frozen commit. Put late result summaries in a private or PR receipt so that reporting does not itself force another commit and another review cycle.

What we ran

These runs came from our lesson labs, on Python 3.12.3 with the standard library only. The review helper was exercised offline: no model or provider was launched, and no claim here is about reviewer quality.

First, a short toy gate that applies the table above, plus its tests:

$ python3 -W error -m unittest -v
...
Ran 8 tests in <t>s

OK

Then three deliberately planted bugs, one at a time: accept a stale commit, ignore the packet hash, ignore the stage. Each one must break the suite:

mutant COMMIT: FAILED (failures=1)
mutant PACKET: FAILED (failures=1)
mutant STAGE: FAILED (failures=1)

Then the real helper at commit 504d930 in a throwaway home directory and scratch repository:

== 1. code-inspection with no expected commit
exit 2
peer_review.py: error: --expected-head is required for code-inspection and PR review
== 2. code-inspection, packet says tests pass but shows no command or exit status
exit 2
Packet lint failed for the code-inspection stage; no provider was launched.
...
verdict_match(pr, 'PLAN_RESULT: ACCEPT') -> False
verdict_match(plan, 'VERDICT: PASS') -> False
verdict_match(plan, 'PLAN_RESULT: ACCEPT') -> True
stage_report(pr, plan marker) raises OSError: reviewer verdict does not match the requested stage

To reproduce the second part, clone the repository, check out 504d930, and call the review helper with a scratch repo and a deliberately weak packet.

Decision framing

This costs a reviewer invocation at each stage and the time to write an evidence packet; we have not measured that cost. It pays for itself when a team merges agent output often enough that "who approved what, and of which commit" can no longer be answered from memory. It is not worth running on a typo or a one-line copy edit, where a focused check and a read of the diff are enough.

Risks and limits

  • The gate verifies bookkeeping. A reviewer can still be wrong, and a packet can still be misleading; the lint cannot prove a test run happened. There is also a known race window between the packet check and the report.
  • Only a fresh PR-stage report on the exact HEAD is sign-off. Plan and inspection acceptances are stage-specific and must not be relabelled as merge approval.
  • In the labs we did not run any real model review, so we make no claim about how well a model reviewer finds defects.
  • Cached or copied approvals are not evidence for new code. If the commit or the packet changes, review again.

FAQ

How many review gates does AI-written code need?
Three is a practical minimum for substantive changes: a plan review before the code exists, a code inspection once the change and its focused tests exist, and a final PR review of the frozen commit with full validation evidence. A small, local, reversible fix can skip the ceremony and rely on a focused check plus a self-review of the diff.
Why must a review be bound to a commit hash?
A review is a statement about specific code. If the report does not name the full commit it examined, a later push can inherit an approval that never covered it. Requiring the reviewed commit to equal the current HEAD makes stale approvals fail automatically.
Can the same model that wrote the code review it?
It can give an early opinion, but it should not be the only gate. Ask a different model, a different agent or a person for the stage approvals, and record which one it was.
Does a passing review mean the code is correct?
No. The gate only proves that a named reviewer accepted a specific commit and packet at a specific stage. It does not prove the reviewer was right, and the packet's test evidence is only as honest as whoever wrote it.

Sources