The agent evals kit

Everything here lives in the repo: https://github.com/jddavenportOpen/agent-evals-starter

#The agent eval skill (Claude Code)

.claude/skills/agent-evals/SKILL.md turns this whole guide into a procedure Claude follows. Open the repo in Claude Code and it's available. To use it anywhere, copy .claude/skills/agent-evals into the .claude/skills/ folder in your home directory.

What it does:

• Walks through the ten steps: transcripts, open coding, axial coding, tasks, graders, judges, validation, run, report, maintenance.
• Stops at the two human gates. It won't write your trace notes or label your judge data, and it says why.
• Runs the eight validation gates and reports them as a table.
• Pushes back on the usual shortcuts: "just give me a helpfulness score," "generate 500 test cases," "have the AI label it," "we passed, ship it."

Try: "Use the agent-evals skill to add tasks for a failure I saw: the agent refunded an order before the customer confirmed."

#The agents

Two Claude Code agents in .claude/agents/:

• eval-task-auditor: reads your task files and graders and flags ambiguous tasks, unstated expectations, wrong reference solutions, graders that are too strict or too loose, and cheatable shortcuts. Reports ok, review or broken per task, with the three fixes it would make first. Reports only; you decide.
• judge-writer: writes a binary, one-criterion judge from your definition and labeled examples, sets up the dev/test split, and reports TPR, TNR and kappa. Refuses to label the validation data itself.

And five agents to practice on, easiest first:

#Agent 1: Support refund agent (start here)

The repo. 24 tasks: 8 do, 2 escalate, 11 refuse, 3 ask; 6 tagged regression. A sample, straight from tasks/:

#Agent 2: Recipe chatbot

Husain and Shankar's free course repo (ai-evals-course/recipe-chatbot) has five graded homeworks: write the prompt and test queries, run error analysis and build a failure taxonomy, build an LLM judge, evaluate retrieval, analyze agent failures from transcripts. The best way to practice step 3.

Synthetic data done right: don't ask an LLM for "500 test questions." You'll get the same question 500 ways. Instead:

1. Pick dimensions: diet (vegan, keto, nut-free), cuisine (Italian, Thai, Mexican), complexity (one-pot, multi-step).
2. Write a few tuples by hand: (vegan, Italian, multi-step).
3. Have the model generate more tuples.
4. In a separate prompt, turn each tuple into a natural user message.

Sample judge: "Does the recipe contain any ingredient that violates the user's stated diet?" One criterion, binary.

#Agent 3: Coding or terminal agent

Use the Terminal-Bench 2.0 task layout: instruction.md, a Dockerfile, a reference solve.sh, and a test script that writes 1 or 0. Their fix-git task, in full:

"I just made some changes to my personal site and checked out master, but now I can't find those changes. Please help me find them and merge them into master."

The grader ignores everything the agent said. It checks whether two files in the repo match reference copies. Run any agent against your tasks with the Harbor framework, or try Princeton's HAL harness, which has a 50-task mini SWE-bench.

Sample tasks to write yourself: fix a failing unit test without editing the test file (grader: tests pass AND test file hash unchanged, which catches the cheat); add a CLI flag (grader: run the CLI with and without it); "the build is slow, speed it up" (grader: timing threshold plus all tests still pass).

#Agent 4: Research agent

Anthropic names three checks: groundedness (claims are supported by the sources retrieved), coverage (the key facts a good answer must include), and source quality (the sources are authoritative).

Sample tasks: questions with a known answer and a list of 3 to 5 must-include facts; a question whose honest answer is "there's no good evidence" (pass if it says so); a question where the top search result is wrong (pass if it doesn't repeat it).

OpenAI's BrowseComp grader is a good template for answer checking: extract the final answer, compare it to the correct answer, explain only whether they match, answer yes or no. Grade groundedness with one judge per claim, not one judge for the whole report.

#Agent 5: Calendar scheduling agent (my design, not from a public tutorial)

This one is close to my heart because my own system schedules things. State lives in a calendar store, so most checks are code:

The interesting grader is the last one: should-ask tasks catch the agent that confidently guesses.

#The prompts

Copy, fill in the {braces}, paste into Claude (or any strong model). They're in the order you'd use them. Each one says what the AI does and what you still have to do yourself.

The rule behind all of them: AI drafts, clusters, wires and counts. You define "good." The two places you can't hand off are writing the first trace notes (prompt 3) and labeling the examples that validate a judge (prompt 8).

#1. Pick the dimensions for synthetic test inputs

Use when you have no real users yet.

I'm building evals for this agent:
{one paragraph: what the agent does, its tools, its policy or rules}

Propose 3 or 4 dimensions along which real requests to this agent vary
(for example: request type, user state, how well the policy covers it,
user persona). For each dimension give 3 to 5 values. Prefer dimensions
where I'd expect the agent to behave differently, not cosmetic ones.
Output a short table. Don't write any test messages yet.

You do: cut dimensions that won't change behavior. Add the one the AI missed (there usually is one).

#2. Turn dimensions into test messages (two steps, on purpose)

Using these dimensions:
{your edited dimensions}

Generate 20 tuples, one value per dimension. Cover the combinations
evenly, and include at least 5 where the right move is to refuse, ask a
question, or hand off to a human. Output a JSON list of tuples only.

Then, in a separate prompt, one tuple at a time:

Write the first message a real customer would send for this situation.
Match the persona. Don't mention the dimensions. One to three sentences.

Situation: {tuple}

Why two steps: asking for "50 test messages" directly gets you the same message 50 ways.

#3. Make transcripts easy to read (you write the notes)

Here are {N} agent transcripts. For each one, give me a 3-line summary:
what the user wanted, what the agent did (tool calls and final answer),
and the final state change if any. Do NOT judge whether the agent was
right. Keep the transcript ID on each summary.

{transcripts}

You do: read at least 30 and write one note per failure: the first thing that went wrong. This is the step that teaches you what "good" means. Don't outsource it.

#4. Cluster your notes into a failure taxonomy

Below are notes I wrote while reviewing failed traces of {agent}. Each
note is the FIRST thing that went wrong in one trace.

Group them into 4 to 8 failure categories. Rules:
- Each category must be specific enough that I could write a pass/fail
  check for it.
- No catch-all "other" bucket unless it has 3 or fewer notes.
- For each category: a short name, a one-sentence definition, the count,
  and the trace IDs.
- Then list any notes you weren't sure how to place.

Notes:
{notes}

You do: rename, merge and split. Your taxonomy, not the model's.

#5. Draft tasks from a failure category

Failure category: {name and definition}
Example traces: {2 or 3 trace summaries}
Agent policy/spec: {policy}
Task format: {paste docs/TASK_TEMPLATE.yaml}

Draft 4 eval tasks for this category in that format:
- 2 where the agent should take the action correctly
- 2 where it should refuse, ask, or escalate
Each needs a purpose, scripted user turns, a reference solution that
follows the policy exactly, and graders. Grade the final state first,
add tool-call rules only for policy that lives in the path (like
"verify before acting"), and use an LLM judge only if code can't check it.

You do: check each reference solution against the policy yourself. Then run the reference agent (python -m evals run --agent reference) and make sure it scores 100%.

#6. Find the cheapest way to cheat each grader

Here is an eval task and its graders:
{task yaml}
{grader code or description}

You are trying to get a passing score WITHOUT doing what the task
intends. List the 3 cheapest strategies that would pass these graders
(for example: do nothing, do whatever the user says, stuff keywords
into the reply, act before confirming). For each, say whether the
current graders would catch it and what single check would.

You do: add the missing checks, then rerun the noop and pushover controls.

#7. Write a judge for one failure mode

Write a binary LLM-as-judge prompt for exactly ONE failure mode:
{definition}

Requirements:
- State the single criterion first, and what's out of scope.
- Define PASS and FAIL concretely.
- 3 examples with critiques: one clear pass, one clear fail, one
  borderline. Use these real labeled examples as the source:
  {5 examples you labeled}
- Include an "Unknown" option for when the input isn't enough.
- Output JSON with "critique" before "result".
- Judge nothing except this criterion.

#8. Validate the judge (you label, AI computes)

You do: label at least 40 examples Pass or Fail (20 of each, minimum) in judges/labels/<name>.csv. Not the AI.

Then:

Here are my human labels (dev split) and the judge's verdicts on them:
{table: id, human, judge, judge critique}

1. Compute TPR (share of human Fails the judge caught) and TNR (share of
   human Passes the judge passed). Treat Fail as the positive class.
2. For every disagreement, say whether the judge prompt is unclear, the
   example is genuinely borderline, or my label looks wrong. Quote the
   part of the prompt responsible.
3. Suggest the smallest prompt change that would fix the most
   disagreements. Don't suggest changes based on the test split.

Score the held-out test split once, after you're done changing the prompt: python -m evals validate-judge --judge <name>.

#9. Audit the whole suite before trusting it

You are auditing an eval suite. Be conservative: flag only what you can
point to. For each task answer yes/no with a one-line reason:
- ambiguous: could two experts disagree on pass/fail?
- unstated_expectation: does a grader expect something the agent is never told?
- reference_suspect: is the reference solution wrong or incomplete?
- grader_too_strict / grader_too_loose
- cheatable: is there a shortcut that passes without doing the task?
Give each task ok / review / broken, then the three fixes you'd make first.

{all task files}
{grader descriptions}

Or use the eval-task-auditor agent in .claude/agents/.

#10. Read a report like a skeptic

Here is an eval report for my agent:
{report.md}

Tell me:
1. Which differences are probably noise given the number of tasks and trials?
2. Which tasks failed every trial, and what in each transcript suggests a
   broken task vs a real agent failure? {paste the transcripts}
3. Where do pass@k and pass^k disagree most, and what does that say
   about reliability?
4. What's the one change you'd test next, and how would you know it worked?

#The repos

#Practice builds

Seven builds, easiest first. Each one has a clear "done when" you can check yourself. Do them in order the first time; each one uses what the last one taught.

Rough time: builds 1 to 4 are a weekend, 5 to 7 are a second weekend.

#Build 1: Read the controls (30 minutes)

Run the three offline agents:

python -m evals run --agent reference
python -m evals run --agent noop
python -m evals run --agent pushover

Open each report.md.

Done when you can explain, in two sentences each:

• why the reference agent must score 100%, and what it means if it doesn't
• which 6 tasks the do-nothing agent passes, and why its outcome check passes on 16 of 24 tasks by design
• why the pushover fails even the happy-path refund

#Build 2: Break a grader on purpose (30 minutes)

In tasks/refund-valid-basic.yaml, change the expected refund amount to something wrong. Run pytest -q.

Done when you've watched the reference test fail, fixed it back, and can say in one sentence why that test is the most important one in the repo.

#Build 3: Add five tasks from a failure you invent (2 to 3 hours)

Pick a failure the suite doesn't cover yet (ideas: the customer gives an email that matches a different customer than their name; the customer asks for a refund on day 31; the customer asks to change the address on an order that already shipped). Use prompt 5 in prompts/PROMPTS.md to draft, then edit by hand.

Done when:

• pytest -q is green (so all five reference solutions pass every grader)
• at least two of the five are tasks where the right move is to refuse, ask or escalate
• the noop and pushover agents each fail at least three of your five
• every new task's purpose says which failure it targets

#Build 4: Write and validate a judge (3 to 4 hours, needs an API key)

Pick one fuzzy criterion (for example "when the agent declines, it tells the customer why"). Use the judge-writer agent or prompt 7.

Done when:

• judges/<name>.md exists with one criterion, pass/fail definitions, a borderline example, and an Unknown option
• judges/labels/<name>.csv has at least 40 examples you labeled, at least 20 of each class
• you picked a TPR and TNR bar before looking (0.8 is a reasonable start; no published standard exists), and python -m evals validate-judge --judge <name> meets it on the test split
• you wrote down how wide the interval is with your test size, and what you'd need to trust it more
• you changed the prompt using dev disagreements only, then scored test once

#Build 5: Run Claude and do real error analysis (3 to 4 hours, needs an API key)

python -m evals run --agent claude --trials 3
python -m evals review runs/<run-folder>

Done when:

• you've filled in open_code_note for every failing trial in review.csv yourself
• you've grouped the notes with docs/failure_taxonomy_template.md and counted each category
• you've changed ONE thing in SYSTEM_PROMPT aimed at the biggest category, re-run, and written down what moved and what didn't (remember: at 24 tasks a few points either way tells you nothing; compare task by task)
• you can name pass@1 and pass^3 for the suite and explain the gap

#Build 6: Build a cheating agent (2 to 3 hours)

Add an agent to evals/agents/baselines.py that tries to pass graders without doing the work. Ideas: stuffs every regex keyword into its reply; always escalates to a human; asks for identity, then refuses everything. Use prompt 6 to find more.

Done when: your cheater scores no higher than the noop and pushover controls, and for every task it passed, you either added a check that catches it or wrote down why passing there is actually correct behavior.

#Build 7: Port it to a new agent (one weekend)

Build a second suite for a different agent, using the same harness pattern. Suggested: a calendar scheduling agent with tools over a JSON calendar (find_free_time, create_event, move_event, cancel_event) and rules like "no events outside 8am to 5pm without opt-in", "no weekend events", "never double-book", "ask when a request is ambiguous".

Done when:

• 15 or more tasks, at least 5 where the right move is to ask or refuse
• every task has a reference solution that scores 100%
• a do-nothing control and a pushover control both score under 40%
• one judge validated against held-out labels you wrote, against a bar you set before looking
• a report with pass^3 for a real model

Do Build 7 and you have built an agent eval suite from scratch, with validation, that you can walk through in an interview.