[Cloud] Document trace filtering in the Console, amctl, and the observer MCP server

versioned_docs/version-cloud/concepts/observability.mdx

@@ -108,4 +108,43 @@ By default every trace an agent produces is exported. At high request rates that
 
 Once traces start flowing in, you can explore them in the AMP Console under your agent's sidebar:
 
-- **OBSERVABILITY → Traces** — search and inspect individual traces by time range or correlation ID; expand a trace to see LLM spans, tool spans, and agent reasoning steps
+- **Observability** → **Traces**: search and inspect individual traces by time range, trace ID, or the filters below. Expand a trace to see LLM spans, tool spans, and agent reasoning steps.
+
+### Filter traces
+
+Agent Manager provides the following trace filters:
+
+| Filter | Matches traces |
+|---|---|
+| Status | That ended in an error, or that succeeded. The status comes from the trace's root span. |
+| Duration | That took at least the given time to complete. |
+| Tokens | That used at least the given number of tokens in total, across all LLM calls. |
+| Span count | That have at least the given number of spans, or steps. |
+| Model | That called a model whose name contains the given text. For example, `gpt-4o` also matches `gpt-4o-mini`. |
+| Conversation ID | That belong to the given conversation, or that have no conversation ID. |
+| Tool | That called a tool whose name contains the given text. |
+| Tool failed | Where a tool call failed. With Tool set, only traces where that tool failed. |
+| MCP server | That used an MCP server whose name contains the given text. |
+
+#### In the Console
+
+1. Open the agent and navigate to **Observability** → **Traces**, then select the environment and time range.
+2. Select **Errors** or **Success** in the **Status** toggle.
+
+    ![Trace list with the Status toggle, filter chips, Go to trace ID, and Advanced Filters above the list](../img/observability/trace-filter-bar.png)
+
+3. Click **Advanced Filters**, set the filters you need, and click **Filter**. Each filter appears as a chip above the list. Remove a chip to clear that filter.
+
+    ![Advanced Filters panel with Tokens at least set to 9k and Tool called set to search_issues](../img/observability/trace-advanced-filters.png)
+
+4. To open a specific trace, enter its ID in **Go to trace ID** and press Enter.
+
+The page URL carries the filters and time range, so you can share a filtered view by copying it.
+
+**Export** downloads the matching traces with full span data, up to the list's page size of 10.
+
+#### When results look incomplete
+
+- **The list stops early.** On a busy agent the search can stop before it covers the whole time range, and the list shows **Searched as far as** a time. Scroll down to load more, or narrow the time range.
+- **Conversation ID, tool, or MCP server filters match nothing.** These filters depend on what the agent's traces record. Auto-instrumented agents record them. For an agent that emits its own spans, see [Making traces filterable](../guides/amp-instrumentation.mdx#making-traces-filterable).
+- **A large trace never matches.** For performance, the **Model** filter only checks traces with up to 100 spans, and the tool and MCP server filters only check traces with up to 200 spans.

versioned_docs/version-cloud/guides/amp-instrumentation.mdx

@@ -186,6 +186,30 @@ A couple of kinds work as a *kind* but don't have their full data card extracted
 - **Rerank** is recognised as a kind only — a rerank span gets the rerank icon, no data card. There's no `RerankData` payload today and no per-kind extractor runs for rerank, so model, query, and results are not surfaced. A future enhancement will add a proper rerank payload. (Note: `rerank` isn't a value OTel GenAI enumerates for `gen_ai.operation.name`; AMP reads it because Cohere's OTel instrumentation and OpenLLMetry emit it. If OTel ratifies a rerank operation later, AMP will align.)
 - **Retriever documents.** The retrieved documents on a retriever span are not extracted in v1 (they aren't extracted for any instrumentation source today). The vectorDB chip, collection, and Top-K render; the document list doesn't.
 
+### Making traces filterable
+
+Auto-instrumentation (platform-hosted agents and `amp-instrument`) already records what the [trace filters](../concepts/observability.mdx#filter-traces) need. If your agent emits its own spans, this is what each filter reads:
+
+| Filter | What the observer reads |
+|---|---|
+| Status | On the root span: status `Error`, `error.type`, an error value in `gen_ai.tool.status`, or a numeric `http.status_code` of 400 or more |
+| Duration, Span count | The trace's start and end times and its number of spans |
+| Tokens | `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` (or `gen_ai.usage.prompt_tokens` and `gen_ai.usage.completion_tokens`). Read from the root span, then the root's first non-LLM child span, then summed over up to 50 LLM spans. |
+| Model | `gen_ai.response.model`, else `gen_ai.request.model`, on LLM spans |
+| Conversation ID | `gen_ai.conversation.id` (or `session.id` and similar session keys). Read from the root span, then the root's first non-LLM child span, then the LLM spans. |
+| Tool | The span name: `execute_tool <tool>`, or a name ending in `.tool`, `.tools`, `.function`, or `.functions` (for example `search_issues.tool`) |
+| Tool failed | Status `Error` on a tool span |
+| MCP server | `serverInfo.name` in the `traceloop.entity.output` of a span named `initialize.mcp` |
+
+The observer recognizes LLM spans by name: `chat`, `chat <model>`, or `<vendor>.chat`. It reads LLM spans only in traces with up to 100 spans, so in a larger trace, model, token, and conversation ID values found only on LLM spans are not seen. Tool filters read the span name, not `gen_ai.tool.name`.
+
+To make a manually instrumented agent's traces filterable:
+
+- Open one root span per request, with every other span inside it.
+- Set `gen_ai.conversation.id` and the token totals on the root span, and set its status to `Error` when the request fails.
+- Name LLM spans `chat <model>` and set the model and token attributes on them, as in [Setting up the exporter](#setting-up-the-exporter).
+- Name tool spans `execute_tool <tool>`, and set their status to `Error` when the call fails.
+
 ### Suppressing prompt and completion content
 
 Prompt and completion content is captured by default. To suppress it, set `TRACELOOP_TRACE_CONTENT=false` in the agent's environment. The variable is read by the Traceloop SDK on the auto path and is documented here for parity — on a fully hand-rolled manual setup that bypasses Traceloop, you control what you put into `gen_ai.input.messages` / `gen_ai.output.messages` directly.