Security tests: two locks on the same door

Security tests: two locks on the same door

Module 5 explained the design. Here are the tests, six of them:

• a call without a token is challenged (401, with the first scope);
• the caller comes from the token: a customer's search over HTTP returns her four tickets;
• a read-only token cannot write: the authorization server grants carol only the read scope, so the client's step-up attempt fails, and no ticket exists afterwards;
• a write with only a read token gets 403 and insufficient_scope naming tickets:write;
• a token issued for another server is refused (401);
• the audit log names people and actions and contains no tokens.

Defence in depth, measured

Two of the defects show why you want both locks. When the service stops enforcing "only staff close tickets" (customer_can_close), no security check fails, because the customer's token has no write scope and never reaches the service. The unit layer catches it. Conversely, when the scope table is wrong (ticket_close needs only the read scope), no unit check fails, because the service is not involved. The security layer catches it. Neither lock proves the other works.

A third case: the service's page-size cap can be removed (no_limit_cap), and a call through the tool with limit=1000 is still refused, because the tool's schema says maximum: 20:

Invalid arguments: arguments.limit must be at most 20, got 1000. Fix the arguments and call `tickets_search` again.

A direct call to the service (the unit check) returned 42 tickets for the same request. The schema hid the bug from every test that went through the protocol. That is a real second lock; it is also a reason the unit layer exists, since without it nothing would ever have noticed the first lock was missing.

Does the suite work? Mutation testing

You have 59 checks that pass. That tells you nothing about whether they would fail if something broke. Mutation testing answers it: break the server on purpose, one defect at a time, run the suite, and see whether anything fails. A defect that no check notices is a hole in the suite.

Here are 25 defects, each switched on by a name. They live in the service, the adapter, the protocol layer, the transport and the authorization layer, and they are my inventions: a sample of plausible mistakes, not a complete list of what can go wrong. Before running anything I wrote down which layers I expected to notice each one.

What the matrix shows.

• 23 of 25 defects were noticed by at least one layer. The control (no defect) failed 0 checks.
• Two escaped everything: tie_order (tickets of equal priority sorted by id instead of by recency) and comment_no_touch (commenting does not update the ticket's "updated" time). No check asserted either. Each is a one-line test (section 12 and Easy exercise 2). The lesson is not that the suite is bad; it is that a suite protects exactly the properties someone thought to write down, and only mutation testing tells you which ones are missing.
• 12 of the 25 were noticed by exactly one layer: five by unit alone (the filter, the SQL, the page cap, the idempotency key, the race), three by conformance alone (a read that writes, a leaked field, an unstable order), one by snapshot alone (the description that lies), one by transport alone (the banner), and two by security alone (the scope table, the audience). If you drop a layer to save time, those are the defects you stop seeing.
• The mean was 1.83 layers per defect. Most defects are noticed by more than one layer; that overlap is the cost of the stack and also its safety margin.
• My predictions were exact for 21 of 25. The four I got wrong were all noticed by more layers than I predicted, never fewer. Three came from the contract layer's error probe, which asserts that each probe call fails: so a defect that makes closing a closed ticket succeed also fails that check, by accident. The fourth is a shared dependency: when the tool list's cursor never ends, every check that needs the list fails, so four layers "noticed". Those are not four independent pieces of evidence.

Closing the escapes

def tie_order_is_most_recent_first(bugs):
    d = app.Env(bugs).desk
    ids = [t["id"] for t in d.search("alice", status="open", limit=20)["tickets"] if t["priority"] == "urgent"]
    assert ids == [11, 4], ids          # 11 was updated after 4

It passes on the correct server and fails on the tie_order defect. The test suite includes this and the comment_no_touch equivalent, to prove that the escapes are closable. They are not among the 59 checks, because I wanted the matrix to show you an honest hole.

Two clients, the same server

A server should work with more than one client. The suite goes further and drives one fixed journey through four doors:

The journey's results, with timestamps removed (they come from a counter that differs by path), are identical through every door. The cost differs:

The legacy protocol costs one more message (the handshake) but fewer bytes, because the modern protocol repeats its metadata on every request. HTTP costs the most the first time because of sign-in; module 5 explained why.

The second client you did not have to write

The official MCP Inspector is the reference developer tool for exactly this job. It ships three clients behind one binary: a web interface, a command-line mode for CI and scripts, and a terminal interface. The command-line mode runs a single request and exits (for example --method tools/list, or --method tools/call --tool-name ... --tool-arg key=value), prints the result as JSON, and uses stable exit codes you can branch on: 3 when the server needs authentication, 4 when it is unreachable, 5 when a tool returned isError: true (so a failed call does not slip through an && chain), and 6 when --strict finds a tool schema that is not portable. A one-line CI gate that lists the tools and fails if a named tool is missing is shown on its page.

That makes the Inspector the natural second client in a pipeline: your own suite covers the rules and the advertised text; the Inspector proves that an independently written client can connect and list and call. Hard exercise 1 runs it against this server's stdio entry point, and I did not run it.

What the example does not do

A test suite is honest about what the server lacks. This one has real gaps, and I would not ship it as it is:

• desk_export is unbounded. With 1,000 tickets, one export is 163,951 characters, which is 48 times a page of search results (3,394 characters). A model that calls it fills its context for one answer. The fix is a limit and a cursor, like search (Medium exercise 2). No check fails today, which is exactly the problem: nothing asserts a ceiling on result size.
• No rate limiting. The specification says servers must rate limit tool invocations, validate all inputs, apply access controls, and sanitize outputs. The example does the second and third, only partly the fourth, and not the first (Medium exercise 3).
• No timeouts, no concurrency limits, no per-request logging. The debugging page's advice: structured logs to stderr for stdio, and your own log aggregation or OpenTelemetry for HTTP, with request ids and timings. (Logging over the protocol, notifications/message, is marked deprecated as of the 2026-07-28 revision on that page.)
• The tool list is static. It never sends listChanged, so the dynamic parts of module 2 are untested.
• No real model. Selection and recovery are scripts.

A checklist for "done"

1. Three layers: rules (no MCP), adapter (words and schemas), transport and identity. Each testable alone.
2. Descriptions say what, when and what comes back; every parameter described; closed sets are enums; schemas closed; results have an outputSchema.
3. Every fixable failure is a tool error with a hint; protocol errors only for malformed or unknown requests; unexpected exceptions become a safe generic message plus a log entry.
4. Seven layers of checks, fresh state per check, all green on a correct server.
5. A snapshot of everything the model reads, changed only in a reviewed commit.
6. Both protocol eras run through the same conformance checks.
7. A real child process and a real socket in the transport layer; nothing but JSON on stdout.
8. Scopes and ownership both tested, and each shown to fail without the other lock.
9. Mutation test your suite; close every escape or write down why it stays open.
10. A second client (the Inspector's command-line mode) in CI, branching on exit codes.
11. Ceilings on result size and call rate, tested.