Measure retrieval before you tune it

1. 01Write 20 questions, each with the file that holds the answer

   wc -l evalset.jsonl

   You pass when: 20 lines. Include five that use different words from the page.

   Check step 1 evalset.jsonl

2. 02Print recall at 4 and the mean reciprocal rank

   Recall at 4: how often the right page is in the four chunks you fetch. Reciprocal rank: 1 if it is first, 1/2 if second, and so on.

   uv run --with requests --with "psycopg[binary]" eval.py evalset.jsonl

   You pass when: Two numbers between 0 and 1, and a list of the questions that missed.

   Check step 2 eval.py

3. 03Change one thing: chunk size, overlap, or a rerank step

   uv run --with requests --with "psycopg[binary]" eval.py evalset.jsonl --label after

   You pass when: The same two numbers for the new setup.

   Check step 3 ingest.pyeval.py

4. 04Write the before and after in a table and keep or revert the change

   cat RESULTS.md

   You pass when: Two rows and a one-line decision. A change that does not move the number is reverted.

   Check step 4 RESULTS.md

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.