Level 9 of 14 · about 4 weeks

Evaluation and observability

Test systems that never give the same answer twice, and see what they do once real users arrive.

Modules

1. 1

   Why agents are hard to test

   non-determinism, long trajectories and noisy scores

   Core 3 parts · about 3 hours

2. 2

   Eval sets from real traces

   turning logs into labelled test cases

   Core 3 parts · about 4 hours

3. 3

   Outcome versus trajectory evals

   scoring the answer and the path taken

   Core 3 parts · about 3 hours

4. 4

   LLM-as-judge

   rubrics, bias and agreement with human labels

   Core 3 parts · about 4 hours

5. 5

   Public agent benchmarks

   what GAIA and coding-agent suites measure and what they miss

   Common 3 parts · about 3 hours

6. 6

   Detecting hallucination

   faithfulness checks and consistency tests

   Core 3 parts · about 4 hours

7. 7

   Regression tests in CI

   a quality gate that fails the build

   Core 3 parts · about 4 hours

8. 8

   Tracing

   OpenTelemetry spans, cost and latency per step

   Core 3 parts · about 4 hours

9. 9

   Safe rollout

   Shadow mode, canaries and A/B tests for agent changes

   Core 3 parts · about 4 hours

10. 10

    Feedback loops

    Turning user signals into better prompts and tools without fooling yourself

    Common 3 parts · about 3 hours