All labs · Lab 45 · advanced · 60 min

Let it improve itself, and prove it

Have a model propose changes to your prompt from its own failures, and accept a change only when held-back data says it helped.

You build: improve.py: proposes one prompt change, scores it, and prints ACCEPT or REJECT with the numbers.

  • Python 3.11 or newer and uv
  • An OpenRouter key (free models are enough)
  • Lab 40 done (the judge)
OpenRouter docs

  1. 01Split 60 graded examples into 40 to learn from and 20 held back

    wc -l dev.jsonl heldout.jsonl

    You pass when: 40 and 20 lines, with no example in both.

  2. 02Score the current prompt on the held-back set three times

    The same prompt scores differently on each run. Until you know by how much, you cannot tell an improvement from luck.

    uv run --with openai improve.py --baseline

    You pass when: Three scores and their spread. Any gain smaller than that spread is noise.

  3. 03Show the improver the failures from the 40 and ask for one prompt change

    uv run --with openai improve.py --propose

    You pass when: A changed prompt saved as a candidate, with one sentence on what it changes.

  4. 04Score the candidate on the held-back 20 and decide

    uv run --with openai improve.py --decide && grep -c heldout improver.log

    You pass when: ACCEPT or REJECT with both scores. The count is 0: the improver never saw the held-back set.

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.

Your computer · Fleet · off

Computers are not set up on this site yet. Run each command on your own machine and compare with "You pass when".