transition2.aiStart a project

Software Factories & AI-Assisted Engineering

Specs With Two Reviewers for AI Agents

A spec an agent can be held to has numbered criteria, a way to verify each, negative cases that point back at them, and two independent reviews tied to the exact text.

Eric MacDougall ·

Answer first

A spec that a coding agent can be held to has five parts: an outcome, a scope, numbered acceptance criteria (AC-1, AC-2) that each say how they will be verified, numbered negative cases (NEG-1) that each name the criteria they attack, and the risks. Get two reviews from two different reviewers, one adversarial and one for acceptance, and bind each review to a hash of the exact spec text so that any edit makes both stale. Then prove each criterion with recorded evidence and have someone other than the author and the reviewers sign off.

What the spec contains

A minimal spec, using a toy export feature (adapted from the lab fixture, so not verbatim output):

# Spec: export the report as CSV

Ticket: example-tracker 42
Author: agent-a (model-a)
Click-test: not-applicable (command line tool, no user interface)

## Outcome
Users can export the report as CSV with a header row.

## Scope
- In: the export command.
- Out: other formats.

## Acceptance criteria
- AC-1: The export writes a header row. Verify by: python3 -m unittest test_export.

## Negative cases
- NEG-1 (AC-1): Exporting a report with no rows must not produce a file without a header.

## Risks
- Large reports: stream rows instead of loading them.

Three rules make this more than a template:

  • Every criterion says how it will be checked. "Verify by:" names a command, a query or a live step. A criterion with none is refused.
  • Every negative case names real criteria. NEG-1 (AC-9) fails when there is no AC-9. This keeps negative cases from drifting into generic worries.
  • Placeholders are refused. A section that still holds the template's <...> text is not a spec.

Our harness adds two rules specific to its setup: every criterion and negative case carries a Source: that traces it to a requirement, and, when the harness policy is set to enforce (as in our setup), a resource-budget section is required. The click-test line must say required or not-applicable with a reason of at least ten characters.

Two reviewers, two roles

The adversarial reviewer tries to break the spec: missing failure cases, unsafe shortcuts, criteria that cannot be verified. The acceptance reviewer asks whether the outcome is clear and the criteria add up to it. They can be different models, a native red-team agent, or a person. The rules we enforce:

  • The two roles must be held by different reviewers. The same reviewer in both is refused.
  • The spec author cannot hold either review role.
  • If a role has several reviews, the latest one wins, so a later BLOCK supersedes an earlier ACCEPT.
  • A review from a model must be a plan-stage report whose packet hash equals the hash of the spec text. If it read some other file, it is refused.

Reviews are bound to a hash of the spec "core", the text above the ## Reviews table. Recording a review appends to the table below the core, so it does not change the hash. Editing anything above it does, and both reviews go stale. That is deliberate: a small edit can reopen a hole the reviewer had already closed.

After the spec is approved

The rest of the flow keeps the spec honest:

  1. Attach the spec and the review reports to the ticket.
  2. As you prove each criterion, record evidence that points at a real test run (a recorded id, commits, redacted logs).
  3. After deploy, an independent click-tester who is not the author, a spec reviewer or an evidence poster tests the deployed build and attaches its own report.
  4. The main agent signs off, and must not be the owner or either reviewer.
  5. A final check lists anything missing before the ticket can close.

For the review stages around the code itself, see Review Gates for AI-Written Code: Plan, Inspection, PR.

What we ran

Native-agent reviews (the harness has an option for reviews by a native agent) were not run. All runs are from our lesson labs, on Python 3.12.3 with the standard library only, in a throwaway home directory and a scratch repository. The real tool is the spec helper from the harness at commit 504d930. Its steps that need a ticket system (new, attach, evidence, click-test, signoff, done) were not run.

A toy spec checker, 11 tests, then seven deliberately planted bugs, each of which must fail the suite:

$ python3 -W error -m unittest -v
...
Ran 11 tests in <t>s

OK
mutant CORE: FAILED (failures=1)
mutant STALE: FAILED (failures=1)
mutant VERIFY: FAILED (failures=1)
mutant NEGREF: FAILED (failures=1)
mutant LATEST: FAILED (failures=1)
mutant SAME: FAILED (failures=1)
mutant AUTHOR: FAILED (failures=1)

The real linter on a flawed spec, then the fixed one:

== 1. lint a spec with a placeholder, a missing Verify by and a NEG naming an unknown AC
exit 2
FAIL
  - section `## Outcome` still holds a template placeholder like <...>
  - AC-1 has no `Verify by: <check>`
  - NEG-1 must name existing criteria in parentheses (unknown: AC-9)
== 2. lint the fixed spec
exit 0
PASS sha256 of the core: <hash16>...

The packet handed to reviewers is the core, and its hash is the core hash. A hand-written fixture report bound to the wrong packet is refused, and the one bound to the right packet is recorded:

packet sha256 == spec core sha256: True
spec-flow: the report reviewed a different packet; run spec_flow.py packet spec.md --out <path> and review exactly that file

Finally the real review_problems check, called directly. The same-person and author-as-reviewer records were built by hand for this check; they are not reviews recorded through the tool:

good.md -> no problems
edited.md -> ['no adversarial review of the current spec (earlier reviews are stale: the spec changed after review)', 'no acceptance review of the current spec (earlier reviews are stale: the spec changed after review)']
same person in both roles -> ['both reviews come from the same reviewer/model; get the other role from a different model, a native red-team agent, or a human']
author reviewing own spec -> ['the adversarial reviewer is the spec author']

The plan report in this run was written by hand as a fixture. It is not a model review.

Decision framing

Writing and reviewing a spec adds time before any code exists, and for a one-line fix it is not worth it. It pays back when an agent will work for hours unattended, when several people or agents touch the same change, or when someone later asks what "done" was supposed to mean. The review and sign-off steps do not make the spec correct; they make it harder for one author, or one reviewer counted twice, to declare it correct.

Risks and limits

  • Our threat model is a careless or hurried agent, not an adversarial one. Reviewer names are self-attested unless verified against a record, and the tool prints unverified names under a "self-attested" heading.
  • A well-formed criterion can still be the wrong criterion. The linter checks structure, not whether the outcome is worth building.
  • Hash binding detects edits to the spec text. It does not detect that the code later drifted from it; that is what the evidence and click-test steps are for.
  • We did not run any real model review in these labs, so we make no claim about how well a model reviewer finds spec gaps.

FAQ

What should a spec for a coding agent contain?
An outcome, the scope in and out, numbered acceptance criteria that each say how they will be verified, numbered negative cases that each name the criteria they attack, and the main risks. Without a verification method, a criterion is a wish.
What is a negative case?
A negative case describes input or behavior that must be refused or must not happen, tied to an acceptance criterion, for example that exporting a report must not write a file outside the chosen folder. It forces the author to think about failure before the code exists.
Why two reviewers instead of one?
One reviewer attacks the spec for gaps and abuse cases, the other checks that it states the outcome and can be accepted. If one reviewer fills both roles, you have one opinion counted twice, so the roles must be held by different reviewers.
Why do reviews go stale when the spec is edited?
Each review is tied to a hash of the spec text it read. Change the text and the hash changes, so the earlier approvals no longer describe the current spec and must be repeated.

Sources