Covers: The experiment: sixteen programs, six levels; Filesystem and network: you need both and more.
The experiment: sixteen programs, six levels
Sixteen small programs: thirteen harmful ones and three that spell harm differently (section 9). Each is judged from outside: whether the canary appeared in the output, whether the honeypot received the key, whether the protected file changed, whether a process with a marker was still alive, how long it ran. A program's own claim about itself is never used.
Red: the harm happened. Green: contained.
Program
What it does
First level at which it stays contained
loop, sleep
never stops, or holds the call open
L1
memory
allocates 600 MB
L1
flood
prints 50 MB
L1 (the output cap)
disk
writes 200 MB
L1 (file-size limit)
env_secret
prints an API key from the environment
L2
cwd_overwrite
overwrites a file in the application's directory
L2
fork, orphan
starts 40 processes; leaves a detached one behind
L3
exfil
reads the key and sends it to the honeypot
L4
file_read, shell_cat, write_outside
reads the key (directly, through a shell command) or plants a file in the home directory
L5
Four things to notice.
Each layer closes what it was built for and nothing else. The environment scrub (L2) does not stop a program from reading the key by path; network isolation (L4) stops the key leaving but not reaching the model, because the program can still print it and the tool returns what it printed. Only the filesystem layer stops that.
At the bare level all sixteen did harm; at L5 none did. That is for this set of programs. It is not a proof about all programs.
Harmless work still works. The experiment also runs three programs that must succeed (arithmetic, a syntax error that must be reported, a division by zero that must be reported) and four ordinary programs (write and read a CSV in the working directory, compute a mean, run echo through a subprocess, fetch a page from a local server). They all gave the right answer at every level, except the last one from L4 on: the fetch needs the network. That is the real cost of the network layer, and the answer in practice is an allowlist proxy (section 10), not an open network.
Each layer costs little. Starting a sandbox, running pass and getting the result takes a median of 23 ms bare and 28 ms at L5 on this machine. Times vary from run to run; the point is the order of magnitude, which is milliseconds.
The question behind this table
"Is it enough to run the code in Docker?" Answer: it depends what is configured. A container is a bundle of exactly these kinds of mechanisms (namespaces, resource limits, a filtered set of system calls), and a container with the default settings is not the same as a container hardened for untrusted code: it shares the host's kernel, may have a network, and may mount directories you did not mean to share. The way to answer is to say what you would configure and test, not which product you would name.
Filesystem and network: you need both
The documentation for Claude Code's own sandbox is blunt about it: effective sandboxing needs both. Without network isolation a compromised agent can send out files such as SSH keys; without filesystem isolation it can tamper with the system to get network access. I tested the first half with two attacks and four configurations. One program reads the fake key and prints it. Another sends the data the agent was given (a file customer.csv that the host put in the working directory) to the honeypot.
Two attacks against four configurations. Only both layers together stop both.
Configuration
Key printed to the model?
Input data sent to another machine?
neither
yes
yes
network only
yes
no
filesystem only
no
yes
both
no
no
The network layer cannot protect what the program is allowed to see, and the filesystem layer cannot protect data that is inside the allowed area. The second attack matters because it needs no secret from outside: the data the user gave the agent is itself the thing worth stealing.
A filter on the code is not a sandbox
The tempting shortcut is to look at the code before running it and refuse anything dangerous. I wrote that filter (it parses the code and flags imports of os, subprocess, socket and others, and uses of open, eval and exec) and measured it on the same programs.
Programs
Flagged
Not flagged
13 harmful
9
4: loop, sleep, memory, flood (nothing in the code looks dangerous; the danger is in how much)
3 harmful programs, spelled differently
1
2: a module imported through importlib with a name built from pieces, and a function fetched with getattr
4 ordinary programs
3 (false alarms)
1: anything that opens a file, runs a subprocess or fetches a page is refused
A filter catches the obvious spelling, misses resource attacks entirely, can be walked around by anyone who writes the same thing differently, and blocks ordinary work. What it is good for is a cheap, early hint to the model ("this sandbox has no network, so do not import socket"). Keep it as a courtesy and let the sandbox be the defence.
The file API: code that runs outside the wall
Data has to get in and out. A tool that accepts "put this file" and "give me that file" runs on the host, with the host's privileges, and the name comes from code running inside. That is a recipe for a confused deputy: a privileged helper that can be steered into using its privilege for someone who has none.
The rules the example implements, each one a check:
A name may not contain .. or be absolute. Obvious, and still the classic.
A name may not lead out through a link the program made. The sandboxed program can create a symbolic link named report.txt that points to the key's path. Inside the sandbox the target does not exist (it is hidden), so the link looks harmless. On the host it points at the real file. If the host reads report.txt, it reads the key. In the experiment: with the check, get refuses ("that name leads outside the working directory"); with the check removed, the secret came back to the host.
Only ordinary files are read, and links are never followed. The last part of the name is opened without following links (even a link to a harmless file inside the directory is refused, my choice), which also closes the gap between checking a name and opening it. A named pipe would make read wait forever, so the type is checked on the open file.
Size limits both ways, a file count, a disk quota, a run count.
Two honest findings from testing this layer. First, the "no .." check and the "no absolute path" check are redundant with the link check, which also refuses those names. Remove either one alone and nothing in the test suite fails: they are equivalent mutants (section 11). Removing all three together fails three checks. Second, a check that looks like it covers one rule may be covered by another, and a suite cannot tell which guard is doing the work unless you take them away one at a time.