All labs · Lab 12 · beginner · 30 min

Pick a model with numbers

Run the same five tasks on three models and choose one from a table of speed, cost and quality.

You build: bench.py and results.csv: 15 rows you scored yourself, and a three-line DECISION.md.

  • Python 3.11 or newer and uv
  • Ollama
  • An OpenRouter key (free models are enough)
OpenRouter docsOllama docs

  1. 01Write five real tasks from your own work, one per line

    Public leaderboards rank models on other people's tasks. Five of your own tell you more than any of them.

    wc -l prompts.jsonl

    You pass when: Five lines. Each is something you would really ask, with an answer you can judge.

  2. 02Write bench.py: every prompt on one local and two hosted models, timed

    uv run --with openai bench.py && column -s, -t results.csv | head -20

    You pass when: 15 rows, each with model, prompt number, seconds, tokens in, tokens out and cost.

  3. 03Score every answer from 1 to 5 without looking at the model name

    Hide the model column while you score. Knowing which model wrote an answer changes how you grade it.

    uv run bench.py --summary

    You pass when: One line per model: average score, average seconds, total cost.

  4. 04Write down your pick and the number that decided it

    cat DECISION.md

    You pass when: Three lines: the model, the reason as a number from your table, and what would make you switch.

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.

Your computer · Fleet · off

Computers are not set up on this site yet. Run each command on your own machine and compare with "You pass when".