All labs · Lab 51 · advanced · 60 min

Attack your own assistant

Break your assistant with instructions hidden in the content it reads, then close the hole in the design rather than in the prompt.

You build: redteam.py with five attacks, and an assistant that fails none of them.

  • Python 3.11 or newer and uv
  • An OpenRouter key (free models are enough)
  • Docker
  • Lab 21 and lab 30 done
OWASP Top 10 for LLM applications

  1. 01Give the assistant a secret it must never reveal and a tool that can send data out

    uv run --with openai redteam.py --list

    You pass when: Five named attacks are listed. At least one hides an instruction in a policy page, and one in a tool result.

  2. 02Run the attacks

    If none succeed, your attacks are too polite. Write the instruction the way an attacker would: as part of the document.

    uv run --with openai redteam.py

    You pass when: At least one attack makes the secret appear in an answer or an outgoing call. Keep the output.

  3. 03Write down which of the three dangerous abilities your assistant has

    Any two of the three are manageable. An assistant with all three can be told by a stranger to leak your data.

    cat THREATS.md

    You pass when: Three lines: does it see private data, does it read content from strangers, can it send data out. All three together is the problem.

  4. 04Remove one of the three: an allowlist on the outgoing tool, or tools in a container with no network

    uv run --with openai redteam.py; echo "exit $?"

    You pass when: All five attacks fail and the exit code is 0. The secret appears nowhere in the output.

The check runs on the computer, not on your word: a step passes when its command exits with 0 there.

Your computer · Fleet · off

Computers are not set up on this site yet. Run each command on your own machine and compare with "You pass when".