Grade with a judge you have checked

1. 01Grade 40 claim decisions yourself: right or wrong

   wc -l labels.csv

   You pass when: 41 lines: a header and 40 rows of claim, decision and your grade. Make about a quarter of them wrong decisions.

   Check step 1 labels.csv

2. 02Write a judge prompt and grade the same 40 with a model

   uv run --with openai --with scikit-learn judge.py labels.csv

   You pass when: A 2 by 2 table of your grade against the judge's, with precision and recall.

   Check step 2 judge.pyJUDGE.md

3. 03Print agreement beyond chance (Cohen's kappa)

   Plain agreement flatters a judge when most answers are right. Kappa takes out the agreement you would get by luck.

   uv run --with openai --with scikit-learn judge.py labels.csv --kappa

   You pass when: A number up to 1. Below 0.6, read the rows you disagree on, tighten the judge prompt and run again.

   Check step 3 JUDGE.md

4. 04Decide which mistake is worse: paying a bad claim or refusing a good one

   cat COSTS.md

   You pass when: A price on each mistake, and the judge setting you chose because of it.

   Check step 4 COSTS.md

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.