Errors that tell the model what to do next: Build and test a server

Covers: Errors that tell the model what to do next; The seven layers of tests; Unit tests: the rules.

Errors that tell the model what to do next

The specification separates two ways to report failure:

Protocol errors (a JSON-RPC error): an unknown tool, a request that does not fit the call schema, a server fault. The page says these are "issues with the request structure itself that models are less likely to be able to fix".
Tool execution errors (a normal result with isError: true): API failures, input validation errors and business-logic errors. The page says these carry "actionable feedback" so the model can correct itself and retry, and that clients should pass them to the model.

So the rule for this server is: anything the caller could fix is an isError result with words in it; only a malformed or unknown request is a protocol error. The conformance suite checks the boundary: an unknown tool gives code -32602; a missing ticket, a closed ticket, a bad argument and a forbidden action are all isError results.

Inside the service, every fixable failure is a DeskError with three parts: a code for tests and logs, a message (what is wrong), and a hint (what to do). The adapter joins message and hint. Examples from the running server:

Situation	Text the model sees
closing a closed ticket	Ticket 5 is already closed (resolution: 'Pointed to Settings > Seats.'). If it needs more work, call `ticket_reopen` with ticket_id 5. Otherwise do nothing.
a ticket that does not exist, or belongs to someone else	No ticket 3 that you can see. Call `tickets_search` to list the tickets you can see.
a customer tries to close	Only support staff can close tickets. Add a comment with `ticket_comment` to ask staff to do it.
a misspelled priority (rejected by the schema)	Invalid arguments: arguments.priority must be one of ['low', 'normal', 'high', 'urgent'], got 'urgentt'. Fix the arguments and call `ticket_create` again.

Two details matter. The second row gives the same answer whether the ticket does not exist or belongs to another customer, so ids cannot be probed (module 5's per-user rule). And an unexpected exception inside a tool (a bug) is caught by the server, logged for the operator with a reference, and shown to the model as a generic, safe message: "The tool failed unexpectedly (reference e-1). Do not retry the same call; tell the user." It never contains a stack trace. A stack trace in a tool result is both useless to the model and a leak.

What the wording is worth

I ran the same eight kinds of failure (a ticket already closed, a number that does not exist, a misspelled priority, a comment that is too long, a customer closing a ticket, an assignee who is not staff, a bad cursor, someone else's ticket) against three versions of the server, with a scripted reader that follows hints it can parse:

The reader recovers from every failure when the server says what to do, and from only the three that the schema itself explains when it does not

8 of 8 recovered with message and hint (mean error text 110 characters, 2.12 calls per task); 3 of 8 with the message alone (68 characters, 1.38 calls); 3 of 8 with only a code (Error: BAD_STATE, 54 characters). The three that recover without any hint are the ones the schema reports, because the schema's own message already lists the allowed values. The five that need the server's words are the business-rule failures.

Warning: what this experiment does and does not show

The reader is a script: it recognises a handful of hint phrasings (callx`,use one of,shorten,leave cursor out`). A real model would recover from some failures that the script cannot, and would sometimes ignore a hint that the script follows. What the experiment shows is which servers give any reader something to act on. It does not show a success rate for a real model. Hard exercise 3 measures that.

The seven layers of tests

Seven layers and 59 checks, on a correct server. Times are for the whole layer on one machine

Layer	Question it answers	Checks	About how long
unit	Are the rules right?	18	80 ms
contract	Do the descriptions, schemas and error texts follow the house rules?	2	5 ms
snapshot	Has anything the model reads changed since someone reviewed it?	1	2 ms
conformance	Does the server follow the protocol, in both protocol eras?	26 (13 x 2)	40 ms
end-to-end	Can a reader choose the right tool and recover from mistakes?	3	20 ms
transport	Does it work as a real child process and on a real socket?	3	850 ms
security	Do tokens, scopes and ownership hold over HTTP?	6	10 ms

Each check is a small function that builds its own fresh help desk, so checks cannot disturb each other, and takes the set of deliberate defects to build it with. On a correct server all 59 pass. The same functions are used again in section 11 against defective servers, which is how the suite is graded.

The times are wall-clock numbers from one machine and mean "roughly this long". The shape is the useful part: the checks that start processes take about five times as long as all the other layers put together, so you run the other six on every save and the transport layer before a release.

Unit tests: the rules

These call Desk directly. They are the largest layer, because the rules are where most bugs are and where tests are cheapest. A few of them are worth reading closely.

Input as data, not as code. The search check passes zzz') OR 1=1 OR (title LIKE ' as the query and expects 0 results. The server's search puts the text in a bound parameter. A version that builds the SQL by pasting the text in returned every ticket for this input. The test is the "attack" itself, which is more convincing than a comment that says the query is safe.

The idempotency key. The check creates twice with one key and expects one row and created true then false. That catches a server that ignores the key. It does not catch a race: two requests with the same key arriving together can both pass a "look first" check and both insert. The race check starts six threads, each with its own database connection to a shared file, and pauses between the look and the write so the race does not depend on luck:

Version	Rows created for one key (6 callers)	Distinct answers
single atomic write with a unique index	1	1
"look first, then insert" with no unique index	6	6

The only real fix is to let the database enforce uniqueness (INSERT ... ON CONFLICT DO NOTHING on a unique index). A unit test around a single caller can never see this, which is why the check uses threads.

Visibility. A customer's search returns exactly her four tickets; asking for someone else's returns the same text as asking for a number that does not exist. And the check on staff-only actions has a second job described in section 10.

Here is one complete check, as written:

@check("unit")
def close_and_reopen_follow_the_state_machine(bugs):
    d = app.Env(bugs).desk
    t = d.close("alice", 1, "Reset the password by hand.")["ticket"]
    assert t["status"] == "closed" and t["resolution"] == "Reset the password by hand."
    expect_error(lambda: d.close("alice", 1, "again"), "BAD_STATE", "`ticket_reopen`")
    assert d.reopen("alice", 1)["ticket"]["resolution"] is None
    expect_error(lambda: d.reopen("alice", 1), "BAD_STATE", "already open")
    expect_error(lambda: d.close("alice", 3, ""), "INVALID", "resolution")