Contract tests and the snapshot

Contract tests and the snapshot

These look at what the server says about itself, with no business logic. They are the cheapest way to protect what the model reads.

Lint for tools

lint.py has thirteen rules about tools and three about error text. Every rule is my house rule, not a protocol requirement (the protocol allows much more). A few examples:

Error text is tested by provoking twelve failures (a closed ticket, a missing one, someone else's, a customer closing, a misspelled priority, a missing customer, a bad cursor, an oversized comment, a non-staff assignee, a forbidden export, a missing argument) and linting each message. The probe also asserts that each one really is an error, which has a useful side effect, discussed in section 11.

The snapshot

The suite stores a fingerprint of everything a client can see about each tool: description, input schema, output schema, annotations. The approved fingerprints live in approved_tools.json. One test compares the running server with them. A change to anything the model reads fails the build until a person approves the new file in a reviewed commit. The failure report names the field:

CHANGED ticket_assign.description

(That report is for the defect "the description now says the tool e-mails the customer, which it does not". No other layer notices that one: it is a true-looking sentence that is false, and only a reviewer reading the diff can say so.) A snapshot test is the build-time version of pinning in module 5: a host that remembers a hash of each tool at approval catches a server that changes a description later; a server that checks its own snapshot catches the change before it ships.

Conformance tests: the protocol's rules, in both eras

These talk to the server through a normal client and also send raw text, because a rule such as "a malformed message gets code -32700" cannot be tested through a client that never sends malformed messages. Each check runs twice, once in the modern protocol (per-request metadata, no handshake) and once in the legacy protocol (initialize first), so there are 13 checks and 26 runs. The checks:

• connect reports the server's name and the tools and resources capabilities.
• tools/list is complete, has unique names, and two listings are identical. The specification asks servers to return tools in a deterministic order, so clients can cache the list and prompt caching works. The check also stops after 20 pages, so a cursor that never ends fails the test instead of hanging it.
• an unknown tool is a protocol error with code -32602; bad arguments and domain failures are tool errors the model can read.
• malformed JSON gets -32700, not a JSON-RPC message gets -32600, and an unknown method gets -32601.
• a notification gets no reply; ids (a number and a string) come back exactly as sent.
• every tool's structured result validates against its outputSchema, and the text content is the same JSON.
• paging a search with next_cursor terminates and covers every ticket exactly once.
• every tool marked read-only changes nothing (compared before and after in the database).
• resources and prompts report unknown names as errors.
• after provoking every failure, nothing unexpected was raised inside a tool (the server's error log is empty).

Because the checks run in both eras on the same server, they also test the claim that the two eras give the same tool list.

End-to-end tests with a stand-in reader

The unit and conformance layers check that the server is right. This layer checks that a reader of the advertised text can use it. The reader is a script (section 3's warning applies). It does two things: choose a tool for a goal, by counting the words the goal shares with each tool's name and description (rare words count for more, so a word that appears in every tool, like "ticket", says nothing); and recover from an error by following the hints in it.

Tool selection. I wrote eight goals, one per tool, such as "mark ticket 7 as solved" or "hand ticket 4 to a staff member":

Recovery. Eight scenarios (section 3), each run through the stand-in with at most three repairs, passing when the final database state is right and no more than five calls were made. All eight pass on the correct server. One further check is a multi-step triage task, run as scripted calls: find open urgent tickets that nobody owns (one: ticket 11), assign it, comment on it, read it back, and check the server logged no unexpected exception.

Transport tests

A server can pass every in-process test and still fail the moment it is started, because the transport adds failure modes that no function call has.

Standard input and output. The stdio_main.py server reads one JSON message per line on stdin and writes one per line on stdout, and writes nothing else on stdout. Logs go to stderr. The debugging page says it plainly: local servers should not log to stdout, because it interferes with the protocol; stderr is captured by the host. The test starts the server as a child process (a real one), runs the full connect, list and call sequence, and checks that stderr received the "ready" log. The defect is the classic one: a friendly start-up banner printed to stdout. What the host sees:

stdout carried a line that is not a JSON-RPC message: 'helpdesk server v1.0 starting up...'

No in-process test can catch that, because the in-process pipe never uses a real standard output. Only a test that starts the real program can.

HTTP on a real socket. The test serves module 5's protected server on 127.0.0.1, sends a real HTTP request with a token, and checks 200, a JSON content type and the structured result; then sends the same request without a token and checks for 401 with a resource_metadata challenge.

The debugging page also lists problems that only appear when a host launches the server: the working directory can be undefined, so use absolute paths; a stdio server inherits only a limited set of environment variables, so credentials must be passed explicitly. A test of "does the server start from a directory it did not expect?" is a cheap addition (Easy exercise 3).