Explain what an automated test is, and why a project needs a pyramid of them.
Write clear tests with pytest, including parametrised tests and fixtures.
Test code that calls the network without making a single real request.
Always cover the failure case of an endpoint, not only the happy path.
Decide what to test first in an agent project, where the model itself is unpredictable.
Why this matters for agents
A model call is slow, costs money and gives a slightly different answer each time. Tests that depend on it are slow, expensive and flaky.
The skill is to make everything around the model predictable and well tested: input checking, parsing, tool functions, routing, retries and error handling. Then you need only a small number of special checks (evaluations, Level 9) for the model's own behaviour. This module teaches the first part.
The problem: how do you know it still works?
You write a function that cuts a long text down to its first 200 words. You try it with one example, it works, and you move on. A week later you change it slightly to handle extra spaces. Does it still work on the first example? On empty text? You would have to remember every case you tried and try them all again by hand.
An automated test is a small program that runs your code with a chosen input and checks the result, for you, as often as you like. You write each check once. After that, every change you make can be checked in a few seconds.
Test
A small function that runs part of your code and fails if the result is not what you expect. The check itself is called an assertion: a statement such as "this must equal that".
Test runner
A tool that finds your tests, runs them all, and reports which passed and which failed. In Python the most popular one is pytest.
There are several kinds of tests, from small to large:
Unit test
Checks one small piece (a function or class) on its own. No network, no database. Runs in milliseconds.
Integration test
Checks that several real pieces work together, such as an API route calling your client code, or your code talking to a real database.
End-to-end test
Runs the whole system as a user would, ideally with real dependencies. Valuable, but slow and fragile, so you keep few of them.
Regression test
A test written after a bug is found, so that the same bug cannot come back unnoticed. ("Regress" means to slip backwards.)
Many fast tests at the bottom, few slow ones at the top.
The pyramid shows the usual balance: many quick unit tests, fewer integration tests, and only a handful of end-to-end tests. The higher up, the slower and more fragile a test is.
Check yourself
What does an assertion do?
Why do you want many unit tests but few end-to-end tests?