All labs · Lab 34 · intermediate · 35 min

Choose and run open models

Measure what a smaller, more compressed copy of a model costs you in quality and gives you in speed and memory.

You build: A table for one model at two compression levels: size, memory, tokens per second, and your score.

  • Ollama
  • About 6 GB of free disk
  • Lab 12 done (your five prompts)
Ollama docs

  1. 01Pull the same model at 4-bit and 8-bit

    Quantising stores each weight in fewer bits. The file shrinks and runs faster; some accuracy is lost. How much is what you measure.

    ollama pull llama3.2:3b-instruct-q4_K_M && ollama pull llama3.2:3b-instruct-q8_0 && ollama list

    You pass when: Two entries for the same model. The 8-bit one is clearly larger on disk.

  2. 02Time both on one prompt

    ollama run llama3.2:3b-instruct-q4_K_M --verbose "Write a haiku about parcels"

    You pass when: After the answer, a block of timings with an eval rate in tokens per second. Run it for both and note the two rates.

  3. 03See what is loaded and how much memory it takes

    ollama ps

    You pass when: The loaded model with its size in memory and whether it runs on the GPU.

  4. 04Run your five prompts from lab 12 on both and score them blind

    cat QUANT.md

    You pass when: A table with size, memory, speed and average score for each, and the one you would ship.

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.

Your computer · Fleet · off

Computers are not set up on this site yet. Run each command on your own machine and compare with "You pass when".