diff --git a/versioned_docs/version-cloud/concepts/observability.mdx b/versioned_docs/version-cloud/concepts/observability.mdx index 7f1c0be..b8fb580 100644 --- a/versioned_docs/version-cloud/concepts/observability.mdx +++ b/versioned_docs/version-cloud/concepts/observability.mdx @@ -108,4 +108,43 @@ By default every trace an agent produces is exported. At high request rates that Once traces start flowing in, you can explore them in the AMP Console under your agent's sidebar: -- **OBSERVABILITY → Traces** — search and inspect individual traces by time range or correlation ID; expand a trace to see LLM spans, tool spans, and agent reasoning steps +- **Observability** → **Traces**: search and inspect individual traces by time range, trace ID, or the filters below. Expand a trace to see LLM spans, tool spans, and agent reasoning steps. + +### Filter traces + +Agent Manager provides the following trace filters: + +| Filter | Matches traces | +|---|---| +| Status | That ended in an error, or that succeeded. The status comes from the trace's root span. | +| Duration | That took at least the given time to complete. | +| Tokens | That used at least the given number of tokens in total, across all LLM calls. | +| Span count | That have at least the given number of spans, or steps. | +| Model | That called a model whose name contains the given text. For example, `gpt-4o` also matches `gpt-4o-mini`. | +| Conversation ID | That belong to the given conversation, or that have no conversation ID. | +| Tool | That called a tool whose name contains the given text. | +| Tool failed | Where a tool call failed. With Tool set, only traces where that tool failed. | +| MCP server | That used an MCP server whose name contains the given text. | + +#### In the Console + +1. Open the agent and navigate to **Observability** → **Traces**, then select the environment and time range. +2. Select **Errors** or **Success** in the **Status** toggle. + + ![Trace list with the Status toggle, filter chips, Go to trace ID, and Advanced Filters above the list](../img/observability/trace-filter-bar.png) + +3. Click **Advanced Filters**, set the filters you need, and click **Filter**. Each filter appears as a chip above the list. Remove a chip to clear that filter. + + ![Advanced Filters panel with Tokens at least set to 9k and Tool called set to search_issues](../img/observability/trace-advanced-filters.png) + +4. To open a specific trace, enter its ID in **Go to trace ID** and press Enter. + +The page URL carries the filters and time range, so you can share a filtered view by copying it. + +**Export** downloads the matching traces with full span data, up to the list's page size of 10. + +#### When results look incomplete + +- **The list stops early.** On a busy agent the search can stop before it covers the whole time range, and the list shows **Searched as far as** a time. Scroll down to load more, or narrow the time range. +- **Conversation ID, tool, or MCP server filters match nothing.** These filters depend on what the agent's traces record. Auto-instrumented agents record them. For an agent that emits its own spans, see [Making traces filterable](../guides/amp-instrumentation.mdx#making-traces-filterable). +- **A large trace never matches.** For performance, the **Model** filter only checks traces with up to 100 spans, and the tool and MCP server filters only check traces with up to 200 spans. diff --git a/versioned_docs/version-cloud/guides/amp-instrumentation.mdx b/versioned_docs/version-cloud/guides/amp-instrumentation.mdx index c70a9b2..8378e63 100644 --- a/versioned_docs/version-cloud/guides/amp-instrumentation.mdx +++ b/versioned_docs/version-cloud/guides/amp-instrumentation.mdx @@ -186,6 +186,30 @@ A couple of kinds work as a *kind* but don't have their full data card extracted - **Rerank** is recognised as a kind only — a rerank span gets the rerank icon, no data card. There's no `RerankData` payload today and no per-kind extractor runs for rerank, so model, query, and results are not surfaced. A future enhancement will add a proper rerank payload. (Note: `rerank` isn't a value OTel GenAI enumerates for `gen_ai.operation.name`; AMP reads it because Cohere's OTel instrumentation and OpenLLMetry emit it. If OTel ratifies a rerank operation later, AMP will align.) - **Retriever documents.** The retrieved documents on a retriever span are not extracted in v1 (they aren't extracted for any instrumentation source today). The vectorDB chip, collection, and Top-K render; the document list doesn't. +### Making traces filterable + +Auto-instrumentation (platform-hosted agents and `amp-instrument`) already records what the [trace filters](../concepts/observability.mdx#filter-traces) need. If your agent emits its own spans, this is what each filter reads: + +| Filter | What the observer reads | +|---|---| +| Status | On the root span: status `Error`, `error.type`, an error value in `gen_ai.tool.status`, or a numeric `http.status_code` of 400 or more | +| Duration, Span count | The trace's start and end times and its number of spans | +| Tokens | `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` (or `gen_ai.usage.prompt_tokens` and `gen_ai.usage.completion_tokens`). Read from the root span, then the root's first non-LLM child span, then summed over up to 50 LLM spans. | +| Model | `gen_ai.response.model`, else `gen_ai.request.model`, on LLM spans | +| Conversation ID | `gen_ai.conversation.id` (or `session.id` and similar session keys). Read from the root span, then the root's first non-LLM child span, then the LLM spans. | +| Tool | The span name: `execute_tool `, or a name ending in `.tool`, `.tools`, `.function`, or `.functions` (for example `search_issues.tool`) | +| Tool failed | Status `Error` on a tool span | +| MCP server | `serverInfo.name` in the `traceloop.entity.output` of a span named `initialize.mcp` | + +The observer recognizes LLM spans by name: `chat`, `chat `, or `.chat`. It reads LLM spans only in traces with up to 100 spans, so in a larger trace, model, token, and conversation ID values found only on LLM spans are not seen. Tool filters read the span name, not `gen_ai.tool.name`. + +To make a manually instrumented agent's traces filterable: + +- Open one root span per request, with every other span inside it. +- Set `gen_ai.conversation.id` and the token totals on the root span, and set its status to `Error` when the request fails. +- Name LLM spans `chat ` and set the model and token attributes on them, as in [Setting up the exporter](#setting-up-the-exporter). +- Name tool spans `execute_tool `, and set their status to `Error` when the call fails. + ### Suppressing prompt and completion content Prompt and completion content is captured by default. To suppress it, set `TRACELOOP_TRACE_CONTENT=false` in the agent's environment. The variable is read by the Traceloop SDK on the auto path and is documented here for parity — on a fully hand-rolled manual setup that bypasses Traceloop, you control what you put into `gen_ai.input.messages` / `gen_ai.output.messages` directly. diff --git a/versioned_docs/version-cloud/img/observability/trace-advanced-filters.png b/versioned_docs/version-cloud/img/observability/trace-advanced-filters.png new file mode 100644 index 0000000..bc25d8f Binary files /dev/null and b/versioned_docs/version-cloud/img/observability/trace-advanced-filters.png differ diff --git a/versioned_docs/version-cloud/img/observability/trace-filter-bar.png b/versioned_docs/version-cloud/img/observability/trace-filter-bar.png new file mode 100644 index 0000000..582581c Binary files /dev/null and b/versioned_docs/version-cloud/img/observability/trace-filter-bar.png differ