# Search Traces This guide will walk you through how to search for traces in MLflow using both the MLflow UI and Python API. This resource will be valuable if you're interested in querying specific traces based on their metadata, tags, execution time, status, or other trace attributes. MLflow's trace search functionality allows you to leverage SQL-like syntax to filter your traces based on a variety of conditions. While the `OR` keyword is not supported, the search functionality is powerful enough to handle complex queries for trace discovery and analysis. important **Local File Store offers only limited search capabilities and can become slow as data volume grows.** As of MLflow 3.6.0, the FileStore is deprecated. We recommend migrating to a SQL-backed store or Databricks for improved performance and more robust search functionality. note **Archived traces and content search**. Archived traces remain viewable in MLflow, but filters that depend on stored span payloads, such as `trace.text` and `span.content`, no longer match once that payload has been archived out of the SQL store. Tag, metadata, status, and other trace-level filters continue to work. See [Archive Traces](/docs/latest/genai/tracing/observe-with-traces/archive-traces.md) for details. ## Search Traces Overview[​](#search-traces-overview "Direct link to Search Traces Overview") When working with MLflow tracing in production environments, you'll often have thousands of traces across different experiments representing various model inferences, LLM calls, or ML pipeline executions. The `search_traces` API helps you find specific traces based on their execution characteristics, metadata, tags, and other attributes - making trace analysis and debugging much more efficient. ## Filtering Traces in the UI[​](#filtering-traces-in-the-ui "Direct link to Filtering Traces in the UI") The UI search supports all the same filter syntax as the API, allowing you to search by: * Trace inputs * Trace attributes: trace name, status, end time, execution time, run\_id * Trace tags and metadata * Trace assessments: feedback or expectations Use the filters dropdown in the MLflow Trace UI to filter traces by various criteria: ![search components](/docs/latest/assets/images/tracing_search_traces-b3a85a28e78491294f0a3c747768a039.png) For example, searching for traces that with ERROR state: ![Search Traces UI](/docs/latest/assets/images/search-traces-on-ui-b8c6e343054a11ab58c527547ee43974.png) Search for trace inputs: ![Search Traces Inputs UI](/docs/latest/assets/images/search-traces-inputs-on-ui-4f7e2a743c22391ec9bb20939949557e.png) Search for trace assessments by key and value: ![Search Traces By Assessments UI](/docs/latest/assets/images/search-traces-assessments-on-ui-e09348c7f5985a54d8efd1ba117488b3.png) ## Search Query Syntax[​](#search-query-syntax "Direct link to Search Query Syntax") The `search_traces` API uses a SQL-like Domain Specific Language (DSL) for querying traces. ### Visual Representation of Search Components:[​](#visual-representation-of-search-components "Direct link to Visual Representation of Search Components:") ![search components](/docs/latest/assets/images/search_syntax-8db1091a58dda699e58cd3846c12cace.png) ### Supported Filters and Comparators[​](#supported-filters-and-comparators "Direct link to Supported Filters and Comparators") | Field Type | Fields | Operators | Examples | | ----------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------- | | **Trace Status** | `trace.status` | `=`, `!=` | trace.status = "OK" | | **Trace Timestamps** | `trace.timestamp_ms`, `trace.execution_time_ms`, `trace.end_time_ms` | `=`, `!=`, `>`, `<`, `>=`, `<=` | trace.end\_time\_ms > 1762408895531 | | **Trace IDs** | `trace.run_id` | `=` | trace.run\_id = "run\_id" | | **String Fields** | `trace.client_request_id`, `trace.name` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE` | trace.name LIKE "%Generate%" | | **Linked Prompts** | `prompt` | `=` (format: `"name/version"`) | prompt = "qa-system-prompt/4" | | **Span Name/Type** | `span.name`, `span.type` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE` | span.type RLIKE "^LLM" | | **Tags** | `tag.` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE`, `IS NULL`, `IS NOT NULL` | tag.key = "value" | | **Metadata** | `metadata.` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE`, `IS NULL`, `IS NOT NULL` | metadata.\`mlflow\.trace.user\` = "user\_123" | | **Feedback** | `feedback.` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE` | feedback.rating = "excellent" | | **Expectations** | `expectation.` | `=`, `!=`, `LIKE`, `ILIKE`, `RLIKE` | expectation.result = "pass" | | **Full Text** (OSS SQLAlchemy store only) | `trace.text` | `LIKE` (with `%` wildcards) | trace.text LIKE "%tell me a story" | **Value Syntax:** * String values must be quoted: `status = 'OK'` * Numeric values don't need quotes: `execution_time_ms > 1000` * Tag and metadata values must be quoted as strings * Full text search must use `LIKE` with `%` wildcards **Pattern Matching Operators:** * `LIKE`: Case-sensitive pattern matching (use `%` for wildcards) * `ILIKE`: Case-insensitive pattern matching (use `%` for wildcards) * `RLIKE`: Regular expression matching ### Example Queries[​](#example-queries "Direct link to Example Queries") #### Full Text Search (OSS SQLAlchemy store only)[​](#full-text-search-oss-sqlalchemy-store-only "Direct link to Full Text Search (OSS SQLAlchemy store only)") Search for any contents existing in your trace. python ``` # Search for traces containing specific text mlflow.search_traces(filter_string="trace.text LIKE '%authentication error%'") # Search for multiple terms mlflow.search_traces(filter_string="trace.text LIKE '%timeout%'") ``` #### Filter by Name[​](#filter-by-name "Direct link to Filter by Name") python ``` # Exact match mlflow.search_traces(filter_string="trace.name = 'predict'") # Pattern matching with LIKE mlflow.search_traces(filter_string="trace.name LIKE '%inference%'") # Case-insensitive pattern matching with ILIKE mlflow.search_traces(filter_string="trace.name ILIKE '%PREDICT%'") # Regular expression matching with RLIKE mlflow.search_traces(filter_string="trace.name RLIKE '^(predict|inference)_[0-9]+'") ``` #### Filter by Status[​](#filter-by-status "Direct link to Filter by Status") python ``` # Get successful traces mlflow.search_traces(filter_string="trace.status = 'OK'") # Get failed traces mlflow.search_traces(filter_string="trace.status = 'ERROR'") # Get in-progress traces mlflow.search_traces(filter_string="trace.status != 'OK'") ``` #### Filter by Execution Time[​](#filter-by-execution-time "Direct link to Filter by Execution Time") python ``` # Find slow traces (> 1 second) mlflow.search_traces(filter_string="trace.execution_time_ms > 1000") # Performance range mlflow.search_traces( filter_string="trace.execution_time_ms >= 200 AND trace.execution_time_ms <= 800" ) # Equal to specific duration mlflow.search_traces(filter_string="trace.execution_time_ms = 500") ``` #### Filter by Timestamp[​](#filter-by-timestamp "Direct link to Filter by Timestamp") python ``` import time # Get traces from last hour timestamp = int(time.time() * 1000) mlflow.search_traces(filter_string=f"trace.timestamp_ms > {timestamp - 3600000}") # Exact timestamp match mlflow.search_traces(filter_string=f"trace.timestamp_ms = {timestamp}") # Timestamp range mlflow.search_traces( filter_string=f"trace.timestamp_ms >= {timestamp - 7200000} AND trace.timestamp_ms <= {timestamp - 3600000}" ) ``` #### Filter by Tags[​](#filter-by-tags "Direct link to Filter by Tags") python ``` # Exact match mlflow.search_traces(filter_string="tag.model_name = 'gpt-4'") # Pattern matching with LIKE (case-sensitive) mlflow.search_traces(filter_string="tag.model_name LIKE 'gpt-%'") # Case-insensitive pattern matching with ILIKE mlflow.search_traces(filter_string="tag.environment ILIKE '%prod%'") # Regular expression matching with RLIKE mlflow.search_traces(filter_string="tag.version RLIKE '^v[0-9]+\\.[0-9]+'") # Find traces where a tag key exists mlflow.search_traces(filter_string="tag.model_name IS NOT NULL") # Find traces where a tag key is missing mlflow.search_traces(filter_string="tag.environment IS NULL") # Combine null checks with other filters mlflow.search_traces(filter_string="tag.environment IS NOT NULL AND tag.model_name = 'gpt-4'") ``` #### Filter by Metadata[​](#filter-by-metadata "Direct link to Filter by Metadata") python ``` # Exact match mlflow.search_traces(filter_string="metadata.`mlflow.trace.user` = 'user_123'") # Find traces where metadata key exists mlflow.search_traces(filter_string="metadata.`mlflow.trace.session` IS NOT NULL") # Find traces where metadata key is missing mlflow.search_traces(filter_string="metadata.region IS NULL") # Combine null checks with other filters mlflow.search_traces(filter_string="metadata.region IS NOT NULL AND metadata.env = 'production'") ``` #### Filter by Run Association[​](#filter-by-run-association "Direct link to Filter by Run Association") python ``` # Find traces associated with a specific run mlflow.search_traces(filter_string="trace.run_id = 'run_id_123456'") ``` #### Filter by Linked Prompts[​](#filter-by-linked-prompts "Direct link to Filter by Linked Prompts") python ``` # Find traces using a specific prompt version mlflow.search_traces(filter_string='prompt = "qa-agent-system-prompt/4"') ``` note The `prompt` filter only supports exact match (`=`) operator with the format `"name/version"`. #### Filter by Span Name and Type[​](#filter-by-span-name-and-type "Direct link to Filter by Span Name and Type") python ``` # Filter by span name mlflow.search_traces(filter_string="span.name = 'llm_call'") # Pattern matching on span name mlflow.search_traces(filter_string="span.name LIKE '%embedding%'") # Filter by span type mlflow.search_traces(filter_string="span.type = 'LLM'") ``` #### Filter by Feedback[​](#filter-by-feedback "Direct link to Filter by Feedback") python ``` # Filter by feedback ratings mlflow.search_traces(filter_string="feedback.rating = 'positive'") # Pattern matching on feedback mlflow.search_traces(filter_string="feedback.user_comment LIKE '%helpful%'") ``` #### Filter by Expectations[​](#filter-by-expectations "Direct link to Filter by Expectations") python ``` # Filter by expectation values mlflow.search_traces(filter_string="expectation.accuracy = 'high'") # Pattern matching on expectations mlflow.search_traces(filter_string="expectation.label ILIKE '%success%'") ``` #### Filter by End Time[​](#filter-by-end-time "Direct link to Filter by End Time") python ``` import time # Get traces that completed in the last hour end_time = int(time.time() * 1000) mlflow.search_traces(filter_string=f"trace.end_time_ms > {end_time - 3600000}") # Find traces that ended within a specific time range mlflow.search_traces( filter_string=f"trace.end_time_ms >= {end_time - 7200000} AND trace.end_time_ms <= {end_time - 3600000}" ) ``` #### Combine Multiple Conditions[​](#combine-multiple-conditions "Direct link to Combine Multiple Conditions") python ``` # Complex query with tags and status mlflow.search_traces(filter_string="trace.status = 'OK' AND tag.importance = 'high'") # Production error analysis with execution time mlflow.search_traces( filter_string=""" tag.environment = 'production' AND trace.status = 'ERROR' AND trace.execution_time_ms > 500 """ ) # Advanced query with span name and feedback mlflow.search_traces( filter_string=""" span.name LIKE '%llm%' AND feedback.rating = 'positive' AND trace.execution_time_ms < 1000 """ ) # Search with pattern matching and time range mlflow.search_traces( filter_string=""" trace.name ILIKE '%inference%' AND trace.timestamp_ms > 1700000000000 AND span.name LIKE '%llm%' """ ) ``` ## Programmatic Search with Python[​](#programmatic-search-with-python "Direct link to Programmatic Search with Python") [`mlflow.search_traces()`](/docs/latest/api_reference/python_api/mlflow.html#mlflow.search_traces) provides convenient trace search functionality: python ``` import mlflow # Basic search with default DataFrame output traces_df = mlflow.search_traces(filter_string="trace.status = 'OK'") # Return as list of Trace objects traces_list = mlflow.search_traces(filter_string="trace.status = 'OK'", return_type="list") ``` note The `return_type` parameter is available in MLflow 2.21.1+. For older versions, use [`mlflow.client.MlflowClient.search_traces()`](/docs/latest/api_reference/python_api/mlflow.client.html#mlflow.client.MlflowClient.search_traces) for list output. ### Return Format[​](#return-format "Direct link to Return Format") #### 1. DataFrame[​](#1-dataframe "Direct link to 1. DataFrame") The `search_traces` API returns a pandas DataFrame by default with the following columns: * MLflow 3.x * MLflow 2.x - `trace_id` - Primary identifier - `trace` - Trace object - `client_request_id` - Client request ID - `state` - Trace state (OK, ERROR, IN\_PROGRESS, STATE\_UNSPECIFIED) - `request_time` - Start time in milliseconds - `execution_duration` - Duration in milliseconds - `inputs` - Input to traced logic - `outputs` - Output of traced logic - `expectations` - A dictionary of ground truth labels annotated on the trace - `trace_metadata` - Key-value metadata - `tags` - Associated tags - `assessments` - List of assessment objects attached on the trace * `request_id` - Primary identifier * `trace` - Trace object * `timestamp_ms` - Start time in milliseconds * `status` - Trace status * `execution_time_ms` - Duration in milliseconds * `request` - Input to traced logic * `response` - Output of traced logic * `request_metadata` - Key-value metadata * `spans` - Spans in trace * `tags` - Associated tags #### 2. List of Trace Objects[​](#2-list-of-trace-objects "Direct link to 2. List of Trace Objects") Alternatively, you can specify `return_type="list"` to get a list of [`mlflow.entities.Trace()`](/docs/latest/api_reference/python_api/mlflow.entities.html#mlflow.entities.Trace) objects instead of a DataFrame. python ``` traces = mlflow.search_traces(filter_string="trace.status = 'OK'", return_type="list") # list[mlflow.entities.Trace] ``` ### Ordering Results[​](#ordering-results "Direct link to Ordering Results") MLflow supports ordering results by the following keys: * `timestamp_ms` (default: DESC) - Trace start time * `execution_time_ms` - Trace duration * `status` - Trace execution status * `request_id` - Trace identifier python ``` # Order by timestamp (most recent first) traces = mlflow.search_traces(order_by=["timestamp_ms DESC"]) # Multiple ordering criteria traces = mlflow.search_traces(order_by=["timestamp_ms DESC", "status ASC"]) ``` ### Pagination[​](#pagination "Direct link to Pagination") [`mlflow.client.MlflowClient.search_traces()`](/docs/latest/api_reference/python_api/mlflow.client.html#mlflow.client.MlflowClient.search_traces) supports pagination: python ``` from mlflow import MlflowClient client = MlflowClient() page_token = None all_traces = [] while True: results = client.search_traces( experiment_ids=["1"], filter_string="status = 'OK'", max_results=100, page_token=page_token, ) all_traces.extend(results) if not results.token: break page_token = results.token print(f"Found {len(all_traces)} total traces") ``` ## Important Notes[​](#important-notes "Direct link to Important Notes") ### MLflow Version Compatibility[​](#mlflow-version-compatibility "Direct link to MLflow Version Compatibility") Schema Changes in MLflow 3 **DataFrame Schema**: The format depends on the MLflow version used to **call** the `search_traces` API, not the version used to log the traces. MLflow 3.x uses different column names than 2.x. ### Performance Tips[​](#performance-tips "Direct link to Performance Tips") 1. **Use timestamp filters** to limit search space 2. **Limit max\_results** for faster queries when ordering 3. **Use pagination** for large result sets ### Backend Considerations[​](#backend-considerations "Direct link to Backend Considerations") * **SQL Store Backend**: Supports the full search syntax documented above, including: * All trace, span, metadata, tag, feedback, and expectation filters * Pattern matching operators (LIKE, ILIKE, RLIKE) * Full text search with `trace.text` * Optimized performance with proper indexing on timestamp * **Local File Store**: Limited search capabilities. May be slower with large datasets. Not recommended, only suitable for storing small number of traces.