transition2.aiStart a project

TEMPLATES · EVALUATION

AI Eval Kit

For engineers and product owners shipping an LLM feature or agent who need a release decision that is better than "it looked fine in the demo". You get a versioned eval set, a scoring rubric and a release gate that you rerun on every model, prompt or data change.

What is inside

  1. Eval plan: scope, risks, metrics, thresholds, sample sizes and owners.
  2. Test cases (CSV): the schema plus 30 starter cases across happy path, edge, refusal and injection.
  3. Rubric: 0 to 2 scoring per criterion with anchored examples, double-scoring guidance and an agreement check.
  4. Release gate: pass rules that use a lower confidence bound on the pass rate, not only the raw rate.
  5. Injection test guide: how to plant harmless marker instructions in the documents, pages and email your system reads.
  6. run_eval.py: a dependency-free Python runner that calls your endpoint, records outputs, and computes the pass rate and a Wilson lower bound.
  7. Results log (CSV): one row per run with model, prompt hash, data version, pass rate, bound and decision.

Format: a zip of Markdown, CSV and one Python file (Python 3, standard library only). You download it straight after payment.

Buy the AI Eval Kit

Choose a licence

Want the AI Governance Pack too? The bundle is US$119 individual or US$590 team.

Bought the AI Operators Field Guide? Both downloads are included free with the course. Open the course in this browser, then use the links on your downloads page.