TEMPLATES · EVALUATION
AI Eval Kit
For engineers and product owners shipping an LLM feature or agent who need a release decision that is better than "it looked fine in the demo". You get a versioned eval set, a scoring rubric and a release gate that you rerun on every model, prompt or data change.
What is inside
- Eval plan: scope, risks, metrics, thresholds, sample sizes and owners.
- Test cases (CSV): the schema plus 30 starter cases across happy path, edge, refusal and injection.
- Rubric: 0 to 2 scoring per criterion with anchored examples, double-scoring guidance and an agreement check.
- Release gate: pass rules that use a lower confidence bound on the pass rate, not only the raw rate.
- Injection test guide: how to plant harmless marker instructions in the documents, pages and email your system reads.
- run_eval.py: a dependency-free Python runner that calls your endpoint, records outputs, and computes the pass rate and a Wilson lower bound.
- Results log (CSV): one row per run with model, prompt hash, data version, pass rate, bound and decision.
Format: a zip of Markdown, CSV and one Python file (Python 3, standard library only). You download it straight after payment.
Buy the AI Eval Kit
Want the AI Governance Pack too? The bundle is US$119 individual or US$590 team.
Bought the AI Operators Field Guide? Both downloads are included free with the course. Open the course in this browser, then use the links on your downloads page.