Part 5 of 6 · 5 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-sandbox, run pytest -q -x (a few minutes), then python demo.py. Open results.json and find the first level at which each harmful program stops doing harm.
Done when: you list, for five programs of your choice, the level that first contains them and the layer responsible, and name the programs that only L5 contained.
Hint
first_closing_level in evaluate.py does the work from the matrix.
Easy 2: Break a limit by hand (40 minutes)
In runner.py, in _limits, change the CPU hard limit from policy.cpu + 1 to policy.cpu, so that soft and hard are equal. Run the CPU check (python -c "import checks; print(checks.run(only={'C7'}))") at L1 and at L5.
Done when: you report the time taken to stop the loop in each case, explain why they differ, and explain why setting only the soft limit would behave differently inside a process-id namespace.
Hint
the soft limit sends a signal; the hard limit kills. Look at _limits in runner.py.
Easy 3: Write the missing program (40 minutes)
Write a seventeenth harmful program that no current level contains, or show with evidence that you cannot. Ideas: reading /proc of the host, writing to /tmp, a very deep recursion, a huge number of small files. Add it to corpus.py with a judge that decides from outside.
Done when: you show the program, its row across the six levels, and say which layer you would add for it (or why the current layers already contain it).
Hint
make the judge check something outside the sandbox: a file in the world, the honeypot's log, or a process by its marker.
Medium
Medium 1: Close a hole the table cannot see (2 hours)
The disk limit is per file (10 MB), not per sandbox. Write a program that writes 500 files of 1 MB each (under the file limit) inside one run. See what the quota does (it is checked between runs). Then enforce a limit during a run: for example, make the monitor measure the working directory every 50 ms and stop the run when it passes a limit.
Done when: a new check fails against the current code and passes with your fix, a new planted defect (monitor off) is caught by it, and you say what the cost is for a program that legitimately writes a lot.
Hint
the monitor loop in runner.run already wakes every 5 ms; add tree_size(workdir) to it, and report killed_for = "disk".
Medium 2: An allowlist for the network (3 hours)
Add a proxy the sandbox can reach and nothing else. For example, a Unix socket mounted into the sandbox that forwards HTTP requests to an allowlist of host names, logs every request, and answers a denied request with a message that says why. Give the sandboxed program a small helper function that uses it.
Done when: the ordinary "fetch a page" program works for an allowed host and fails with a clear message for a denied one; the honeypot on the loopback still receives nothing; and you describe one way a program could still exfiltrate data through the allowed host (the documentation's warning about broad domains).
Hint
the sandbox's mount namespace can bind one socket file; the proxy runs on the host and checks the Host header against the list.
Medium 3: Prove it on someone else's tool (2 hours)
Replace the tool wrapper with a different reply format of your own design (for example, always JSON). Run the four agent tasks against it, and write two new checks for the reply (for instance: the reply always names the reason for a stop; the reply never exceeds 10,000 characters).
Done when: your reply still lets the scripted agent solve four of four, the new checks pass and fail on a defect that you plant, and you explain what the scripted agent would need to change to read your format.
Hint
keep structuredContent; the scripted agent only reads the text, so either keep the phrases or change the agent.
Hard
Hard 1: Hardened container as another door (4 hours)
Run the same code in a container hardened with the flags in Anthropic's deployment guide (no network, read-only root, a memory limit, a process limit, no capabilities, a non-root user). Re-run the sixteen programs through docker run. (I did not run this, so your results are the ground truth.)
Done when: you give the row for each program, say which of the six layers the container flags correspond to, say which programs behave differently from L5 and why, and state what a container gives you that this sandbox does not (and the reverse).
Hint
--network none --read-only --tmpfs /work --memory 256m --pids-limit 16 --cap-drop ALL --user 1000:1000 is a good start; mount the work directory read-write only for the working files.
Hard 2: A real model on the four tasks (4 hours)
Replace the scripted agent with a real model, using both tools, ten runs per task, and tasks written by someone who did not read the tool's replies. (I did not run this.)
Done when: you report success per task per tool with the model's name and the date, say how often the model recovered from the naive tool's empty reply, and say which of the scripted results held.
Hint
write the replies you expect the model to need before you run, and keep a fresh sandbox per run.
Hard 3: An attack on the harness (5 hours)
Look for a way to harm the host side of this example: the monitor, the output capture, the file API, the listing of files. Write three attacks as programs and try them. (I did not run this.)
Done when: you report each attack's result with evidence, fix the code for any that worked, add a check and a planted defect for it, and say which attack surprised you.
Hint
think about what the monitor reads from the program (process names, file names) and what the file API trusts.