All labs · Lab 23 · intermediate · 45 min

Build a dataset and a baseline to beat

Make a small labelled dataset and score two cheap baselines on it, so later work has a number to beat.

You build: data/ with train, validation and test files, and two accuracy numbers on the same test rows.

  • Python 3.11 or newer and uv
  • An OpenRouter key (free models are enough)
scikit-learn user guide

  1. 01Collect 300 claim emails labelled lost, damaged or late

    Reading your own data is the step people skip and the one that pays most. Note how many of your 30 were wrong.

    wc -l data/all.jsonl

    You pass when: 300 lines of text and label. Have a model write them if you have none, then read 30 yourself and fix wrong labels.

  2. 02Shuffle once with a fixed seed and split 240, 30, 30

    wc -l data/train.jsonl data/valid.jsonl data/test.jsonl

    You pass when: 240, 30 and 30 lines. No email appears in two files.

  3. 03Train word counts plus logistic regression and score it on the test rows

    uv run --with scikit-learn baseline.py

    You pass when: An accuracy number and a 3 by 3 table of which label was mistaken for which.

  4. 04Score a hosted model on the same 30 rows with no training

    uv run --with openai baseline.py --llm

    You pass when: A second accuracy number and its cost. Write both in RESULTS.md: this is the bar.

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.

Your computer · Fleet · off

Computers are not set up on this site yet. Run each command on your own machine and compare with "You pass when".