Tracing and observability

A different question than uptime

Conventional service observability asks whether the system is up, fast, and free of errors. An LLM feature can score perfectly on all three measures and still be entirely useless in practice — confidently answering from the wrong document, drifting off the intended tone, or quietly abstaining on questions it should actually be able to handle. None of this shows up in a latency histogram or an HTTP error rate, because every one of these failures returns a successful response with fluent, well-formed text. The signals that actually matter here are different in kind: what was retrieved for a given query and whether it genuinely contained the answer, what exactly was present in the context window at generation time, which tools were called and precisely what they returned, what the model actually produced at each intermediate step of a multi-step run, how many tokens and how much money a given run consumed, and whether the user who received the result accepted it or immediately tried again.

OpenTelemetry conventions for generative AI

A trace represents one complete run as a tree of nested spans — a top-level span for the overall request, with child spans beneath it for each model call, each retrieval, each tool execution, and each sub-agent invocation, nested exactly where each one logically belongs in the overall flow. Standard semantic conventions have emerged for naming these spans and their attributes consistently — a model-call span typically carries the model name and parameters used, the full rendered prompt and response, and token counts broken out by input and output; a retrieval span carries the query issued and the ids and relevance scores of whatever was returned; a tool span carries the arguments passed in and the raw result returned. Adopting these shared conventions rather than inventing a bespoke logging format means an LLM-specific trace can live inside the exact same observability system already used for the rest of a service’s infrastructure, rather than requiring an entirely separate, parallel tool just for this one part of the stack.

Sampling at volume, and what it costs you

A full trace, carrying entire prompts and responses verbatim, can run to many kilobytes or more per single run — multiple orders of magnitude larger than a typical HTTP access log line — which makes capturing every single production run expensive at real volume. Standard sampling strategies built for ordinary web traffic, such as keeping a fixed small percentage of all requests at random, can quietly discard exactly the rare failure cases that matter most for debugging, since a rare failure mode by definition appears in only a small fraction of traffic and a random sample may simply miss it entirely. A better approach for LLM systems samples adaptively: always keep traces that hit an error, that received explicit negative user feedback, or that triggered a guardrail, while sampling only a smaller baseline fraction of the otherwise unremarkable, successful runs.

Cost attribution, concretely

Because every span already carries token counts and a known price per token, a complete trace can be rolled up into an actual dollar cost for that specific run, and that cost can then be attributed further — by which user made the request, by which feature or endpoint handled it, by which specific model was called. This turns “our LLM costs rose last week” from a vague, hard-to-act-on observation into a specific, answerable question: which feature’s traces grew in volume, length, or model tier, and is that growth something to welcome or something to fix.

From trace to eval: the loop that closes

A production trace is not only useful for debugging a single incident after the fact. It is a real input with a real, observed outcome attached, which is exactly the raw material the eval set is built from. Treating tracing as the primary feeder for evaluation — regularly pulling real traces, especially the ones with negative outcomes, directly into the eval set as new test cases — closes the loop between what actually happens in production and what gets measured going forward. Teams that use tracing this way tend to improve steadily over time; teams that treat tracing only as a dashboard to glance at occasionally during an incident do not see the same compounding benefit.

A debugging narrative, to make this concrete

A user reports that an agent gave a wrong answer about their account balance. Opening the trace for that specific run shows the retrieval span first: it returned the correct account document, so retrieval was not the problem. The next span shows the context assembly step included that document alongside four other unrelated documents from an overly broad initial search, pushing the correct document into the middle of a long prompt. The final generation span shows the model’s answer drawing a number from one of the unrelated documents instead of the correct one — a direct, visible instance of the lost-in-the-middle effect covered in the context window lesson, now caught concretely in a real trace rather than inferred abstractly from research. The fix that follows directly from this trace is tightening the retrieval and re-ranking step so fewer, more precisely relevant documents reach the prompt — a fix that would have been close to impossible to arrive at correctly without seeing the full span-by-span trace of exactly what the model was given at each stage.