Population: 3.9 percent of runs contained a tool error; 0.5 percent were more than three times the median duration; mean cost $0.0245 per run.
Sampling:
Policy
Traces stored
Error traces kept
Slow traces kept
Head, 1%
1.0%
1%
0%
Head, 10%
9.9%
9%
14%
Tail rule (errors, over 3x median, 1% base)
5.4%
100%
100%
Share of traces stored and of error traces retained under head and tail sampling.
Model calls dominate latency and cost in a typical agent run, so per-span attributes tell you where to optimise.
Head sampling is cheap and blind: it keeps the rare failures at the same rate as everything else.
Tail sampling needs the whole trace buffered before deciding, which costs memory and a collector component, but keeps what you need to debug.
Roll-ups make cost a first-class signal: per run, per user, per tool.
Design rules
One trace per user request, with a span per model call, tool call, retrieval and handoff; carry the trace id across services and agents.
Use standard attribute names where they exist, and record the convention version you used because names are still changing.
Record tokens, model, tool name, status and error type; record prompt and output content only on purpose, redacted, with a short retention.
Link evals to traces: store the trace id with each eval result (module 7), so a red build shows the run.
Sample by outcome: keep errors, slow runs, low judge scores, flagged hallucinations and a random base rate.
Alert on aggregates (error rate, p95 latency, cost per run) and use traces to explain them.
Propagate context across async work and between agents, or the tree breaks.
Measure the overhead of your own instrumentation and export asynchronously in batches.
Interview questions, with the follow-ups
How would you debug an agent that fails in production?
"From its trace: a tree of spans for the run with model calls, tool calls, arguments, errors and timings. I find the first span that went wrong, look at its inputs and the previous step's output, then reproduce it with recorded tool outputs and add it as an eval case. With OpenTelemetry's Gen-AI conventions the attribute names are standard, so the same tooling works across frameworks, although the conventions are still in development." Follow-up: "What if it is not in your traces?" Then my sampling dropped it, which is why I use tail sampling that keeps errors, slow runs and low-scoring runs.
How do you track cost per request?
"Record token counts on each model-call span and roll them up over the trace, then multiply by the price table, per user or feature. In my simulation the example run was $0.027 and 90 percent of its time was in model calls. I alert on cost per run and on the long tail, not just the mean." Follow-up: "Where do cached tokens go?" As separate attributes with their own price (Level 7 module 7), or the cost will be wrong.
Head or tail sampling?
"Head sampling is simple and cheap but random, so it keeps 1 percent of failures if it keeps 1 percent of traces, as in my simulation. Tail sampling decides after the trace completes, so I can keep all errors and slow runs for 5 percent of storage, at the price of buffering traces and running a collector." Follow-up: "What if volume is huge?" Combine them: sample the boring majority at the head, and use the tail rule for the interesting ones, accepting the buffering cost.
What should not go in a trace?
"Secrets, and by default raw prompts and outputs with personal data. The conventions treat message content as opt-in. I redact or hash sensitive fields, limit retention, and restrict access, because traces are a copy of user data that engineers can read." Follow-up: "How do you debug without content?" Capture content for a sampled or flagged subset with consent and a short retention, or log structured fields instead of free text.
Common mistakes
Logs instead of traces, so steps cannot be tied to a run.
Head-sampling at 1 percent and then missing every rare failure.
No token or cost attributes.
Capturing full prompts with personal data and no retention.
Broken context propagation, so tool calls appear as separate traces.
Hard-coding attribute names from a draft convention without a version note.
Tracing only the model call, not tools and retrieval.
No link between eval results and traces.
What the example does not do
No OpenTelemetry SDK, exporter, collector or backend; the tracer is minimal and synchronous.
A simulated agent with simple durations; real runs have retries, parallel calls and streaming.
Self time assumes sequential children; parallel spans need the critical-path calculation.
The tail rule is evaluated offline on complete traces; a real collector must buffer.
The convention status comes from a secondary source; I did not read the specification.
Optional real experiment (I did not run this): instrument a real agent with the OpenTelemetry SDK, export to a local backend, and compare your measured per-span overhead and attribute names with this module.
Check yourself
A run was slow. How do you find out whether the model or a tool was the problem? (Read the trace: the span tree shows start, end and parent for each model and tool call. In the simulated run, model calls were 2.94 s of 3.28 s.)
For the curious Sampling decides what you can debug later
Over 20,000 simulated runs, 3.9 percent had a tool error. Random head sampling at 1 percent kept only 1 percent of those error traces; deciding after the run (a tail rule) can keep the errors you need. The OpenTelemetry Gen-AI conventions were still marked Development in the secondary source I read, so record which version of the names you used.
What you will use daily at work
Practice
What it looks like
One trace per request
Spans for each model call, tool call, retrieval and handoff.
Record tokens, model, tool and error type
Content only on purpose, redacted, with a short retention.
Keep error traces
Sample after the run, not only at the start.
Store the trace id with each eval result
A red build then points to a run.
Draft: confirm the current OpenTelemetry Gen-AI attribute names and status, which my tools could not read directly.
What you must be able to say (one page)
A trace is a tree of timed spans describing one run. For an agent: a root span for the agent invocation, child spans for each model call and each tool call, each with start, end, parent, status and attributes (model, tool name, token counts, error type). Traces answer "what happened in this run?"; metrics and logs answer aggregate and detailed questions.
OpenTelemetry (OTel) is the vendor-neutral standard. It is a CNCF framework for traces, metrics and logs that lets you instrument once and export to any backend, with Gen-AI semantic conventions: standard attribute names and span structures for model calls (model, token usage), tool executions and agent workflows, so traces are portable across frameworks.
Current status, with a caveat. The OTel page I tried to read had moved to a separate semantic-conventions-genai repository, which my tools could not open. A third-party article dated mid-July 2026 says every gen_ai.* attribute and span still carries the "Development" status, names can change (examples it lists: gen_ai.system became gen_ai.provider.name; prompt and completion token attributes became input_tokens and output_tokens), operation names include chat, execute_tool, invoke_agent and invoke_workflow, and message content is opt-in, not captured by default. Verify against the specification before you rely on a name.
Per-step cost and latency fall out of a trace. In my simulated run (3 model calls, 2 tool calls), the whole run took 3.28 s, of which model calls were 2.94 s (about 90 percent) and tool calls 0.34 s; token counts summed to 5,700 input tokens and about 630 output tokens, which at an assumed price is $0.027. "Where does the time go?" is the first question a trace answers.
Sampling is a decision about what you can debug later. Over 20,000 simulated runs, 3.9 percent contained a tool error. Head sampling (decide at the start, random) at 1 percent kept 1 percent of the error traces; at 10 percent kept 9 percent. A tail rule (decide after the run: keep errors, runs over 3 times the median, and 1 percent of the rest) kept 100 percent of errors while storing 5.4 percent of all traces.
Instrumentation is cheap but not free. My minimal tracer measured about 2.3 microseconds per span on this machine with no export; a production SDK with batching and export costs more, which I did not measure.
Privacy: prompts and tool outputs contain personal data; capture content only on purpose, redact it, and set retention (Level 10).
Honest limits: simulated agent, simulated durations and prices, a home-made tracer rather than the OTel SDK, and the status of the conventions comes from a secondary source.