When a user reports that the assistant gave a wrong answer yesterday afternoon, someone has to find out why. Was it retrieval? A tool timing out? A prompt version deployed that morning? A model fallback that kicked in under load?
Without traces, that investigation is guesswork. With them, it is a lookup.
Evaluation tells you how a system behaves on cases you prepared. Observability tells you what it did on the requests real users sent.
A trace is the story of one request
A trace records one end-to-end request. Inside it, spans record each step:
- The incoming request and the user or tenant it belongs to
- Retrieval queries, with the documents returned and their scores
- Each model call, with model name, prompt version, parameters and token counts
- Tool calls, their inputs, outputs, errors and retries
- Guardrail checks and their results
- The final response returned to the user
For agents, spans nest. A planning step spawns tool calls, which may spawn further model calls. The trace should show that tree, not a flat list of log lines.
Open standards such as OpenTelemetry now include conventions for generative AI. Using them keeps AI traces alongside the rest of your service telemetry instead of in a separate silo.
Decide what to capture, and what not to
Full prompts and responses are the most useful debugging data and the most sensitive. Decide deliberately:
- Capture metadata on every request: IDs, versions, timings, token counts, status
- Capture full content on a sampled basis, or only where policy allows
- Redact or mask personal data before it reaches the telemetry store
- Set retention to match your data policy, not the tool’s default
- Restrict who can read content-level traces, and log that access
Cost and latency per span
Aggregate dashboards tell you the bill went up. Span-level telemetry tells you why.
Track for each step:
- Input and output tokens and the resulting cost
- Latency, including time to first token for streamed responses
- Retries and fallbacks that silently double the work
- Cache hits where prompt or response caching applies
Then slice by feature, tenant, prompt version and model. Common findings are mundane: a retrieval step pulling far more context than needed, an agent loop that retries a failing tool, a long system prompt sent on every turn. Each is cheap to fix once you can see it.
User feedback is a signal, not a score
Explicit feedback, such as thumbs up or down and short comments, is valuable but sparse and skewed. People mostly respond when something annoys them.
Implicit signals are often richer:
- The user rephrases the same question immediately
- They copy the answer, or ignore it
- They abandon the flow or escalate to a human
- They edit a generated draft heavily before sending
- They override a recommendation
Attach each signal to the trace that produced the response. A thumbs-down without a trace tells you someone was unhappy. With a trace, it tells you which retrieval result or tool call let them down.
From traces to action
Telemetry only matters if someone looks at it and something changes.
- Review a sample of negative-feedback traces every week with product and domain owners
- Turn confirmed failures into test cases for the evaluation suite
- Alert on shifts in latency, cost per request, error rate and feedback rate by feature
- Watch for changes in input patterns, such as new topics or languages, that the system was not designed for
- Link traces to prompt and model versions so regressions point to a change
Where to start
- Instrument one production flow end to end with spans for every model, retrieval and tool step
- Record model, prompt version, tokens and latency on every span
- Agree content capture and redaction rules with your privacy team
- Wire explicit and implicit feedback to trace IDs
- Hold a weekly trace review and track what it changes
Incubics instruments AI systems for tracing and feedback from the first release, so production questions get answered from data rather than memory.