Hi all, No objections came in, so I've opened the umbrella: SOLR-18571 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1792801902707000&sa=E>, with the full design attached and the five sub-tasks from my last mail. I'm starting on SOLR-11994 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1792801902707000&sa=E>, the core hook.
Feedback is still welcome on the umbrella or on any PR as they come up. Thanks, Amrit On Sun, Oct 18, 2026 09:53 AM, Amrit Sarkar <[email protected]> wrote: > Hi all, > > Bumping this before I start on the code. The only reply so far was Eric's > idea about load-based trade-offs > <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp>. That > stays out of this proposal, but the two connect; more on that below. Nobody > has raised concerns about the plan itself, and I want to be sure that > silence means agreement and not that people missed it. > > *The use case I didn't cover.* A lot of teams at my workplace now > diagnose production issues with agents: Slack bots and Claude-style agents > that call observability tools through MCP > <https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro> and > CLI. Someone asks, "Why was the ingress service slow at 14:05?" and the > agent pulls traces, metrics and logs. > > For Solr, the only source today is solr_slow_requests.log > <https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries> > shipped to a log store. The agent can query that through something like the > Splunk > MCP server > <https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools>, > but its search tool gives up after one minute or 1,000 events, and the > lines are only there if they fit in your daily license volume > <https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works>. > The agent ends up scanning raw log lines to rebuild something Solr already > had in memory when the request finished. > > *How this API closes the gap*, without Solr shipping anything > agent-specific: > > - Applications can already pass their own request ID to Solr through > the X-Trace-Id header > > <https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation> > (solr.traceIdHeader), or traceparent when the OpenTelemetry module > > <https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module> > is on. Solr uses it as the trace ID, and every record carries it. > - An agent can go from the application's request ID to the Solr query, > its latency, CPU, hits and shard count in one call, straight from the > cluster. > - It's a v2 API, so it ends up in the generated OpenAPI spec and SolrJ > client <https://github.com/apache/solr/blob/main/dev-docs/apis.adoc>. > Wrapping it as an MCP tool is a few lines in whatever MCP server a team > already runs. Solr itself stays out of MCP. > > To make that lookup cheap, I'm adding optional collection and traceId > filters to GET /api/node/query-insights/top-queries. Everything else > stays as described in the first mail > <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd>. > > How this ties into Eric's idea. Shedding LTR or vector work under load needs > to know which queries are expensive and what they ran. So each record > will also note the costly features used (LTR, knn, grouping, faceting, > paramset). The load control itself is a separate discussion. > > *How I'll deliver it.* One umbrella JIRA for Query Insights, then small > sub-tasks, each its own PR that can be reviewed and merged on its own: > > 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994>: the > SearchRequestListener hook in core. No behaviour change when nothing > is registered. > 2. The query-insights module: per-node top-N by latency and CPU, the > features each query used, plus the node-local API. > 3. Cluster-wide view: nodes=all merging, with nodes that didn't answer > listed. > 4. collection and traceId filters, and runtime settings through > cluster properties. > 5. Ref guide page. > > Stored history and grouping get their own sub-tasks later, after the first > five land. > > *Two direct questions:* > > 1. Any objections to the plan as described? > 2. Do you think this is a good fit for Solr at all? If you have > doubts, I'd really value hearing them before I start. I'd rather adjust the > idea now than spend reviewers' time on something the project doesn't want. > > If nothing comes up by Wednesday, Oct 21, I'll create the umbrella and > the sub-tasks, post the full design there, and start with the core hook. > I'm not a committer, so I'll also need someone to shepherd it. Eric, would > you be up for that? > > Thanks, > Amrit > > On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar <[email protected]> > wrote: > >> Resending, the first copy seems stuck in moderation. >> >> Hi all, >> >> I've been away from Solr for a while. Coming back, I spent some time >> digging into the observability side, both what Solr exposes today and how >> other search engines handle it, to rebuild my understanding before >> proposing anything. One gap stood out, and I'd like to pick up SOLR-11994 >> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E> >> ("Expose a slow query API", open since 2018) to close it. I want to check >> the shape with you before writing code. >> >> *The problem.* When someone asks "which queries are hurting this cluster >> right now?", Solr has no direct answer: >> >> - slowQueryThresholdMillis >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E> >> writes to solr_slow_requests.log, which you have to ship somewhere >> and grep. >> - Request handler metrics >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E> >> give latency per handler, not per query. >> - Tracing >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E> >> shows individual requests, if you run a backend and sample the right ones. >> - /tasks/list >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E> >> shows only queries that are still running, and only those sent with >> canCancel=true. >> >> None of them gives a ranked list of the most expensive queries. >> >> OpenSearch solved this with its Query Insights >> <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E> >> plugin (2.12+). Each node keeps an in-memory top-N of the searches it >> coordinated >> <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, >> ranked by latency, CPU or memory, over a rolling window. One API call >> returns them from all nodes. It runs inside the OpenSearch process >> <https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E>, >> with no sidecar. >> >> *The proposal.* A new module, solr/modules/query-insights, modelled on >> that plugin: >> >> - *Off by default, two steps to turn on.* List the module in >> SOLR_MODULES >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E> >> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is >> loaded. >> - *Small hook in core.* A SearchRequestListener interface, called at >> the end of SearchHandler >> >> <https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E> >> request handling on the coordinator only (shard sub-requests are skipped). >> With no listener registered, the cost is one isEmpty() check per >> search. >> - *Recording.* Per node, a top-N per metric (latency, CPU) for the >> current and previous clock-aligned window. Defaults follow OpenSearch: >> N=10 >> (max 100), 5-minute windows. A request that doesn't beat the current Nth >> value costs a field read and nothing else. >> - *Records.* Collection, handler, params (redactable, truncated at 8 >> KB), hits, shard count, user, latency, CPU, and the trace id >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>. >> The trace id is something OpenSearch doesn't have: it links each entry to >> the request log and to traces. >> - *API.* GET /api/node/query-insights/top-queries?type=latency|cpu& >> nodes=all, needs the metrics-read >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E> >> permission. Merges across nodes and lists any node that didn't answer. >> - *Settings.* System properties for defaults, ext.queryInsights.* cluster >> properties >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E> >> for runtime changes. >> - *Not in the first PR.* Grouping similar queries by shape >> >> <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, >> memory as a metric, and stored history. History would come later as a Time >> Routed Alias >> >> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E> >> with autoDeleteAge, which gives us retention without a custom cleanup >> job. >> >> *Why not tracing?* Tracing and this work together rather than compete. >> Tracing answers "what happened inside this request?" once you know which >> one to look at. This answers "which requests should I look at?" without a >> tracing backend, and the trace id in each record takes you straight to the >> trace. >> >> *Questions for you:* >> >> 1. Module plus a small core listener, or would you rather see this in >> core? >> 2. Is metrics-read the right permission for data that includes query >> text, given the redaction option? >> 3. Anyone already working on something similar? I found SOLR-1101 >> >> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E>, >> SOLR-11994 >> >> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E> >> and Eric's User Behaviour Insights PR (#2452) >> >> <https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E>. >> I'd like to line up with UBI when we get to stored history. >> >> I've written up a fuller design covering the core hook, recording, >> settings, security and tests, and I'll share it on the JIRA. Feedback on >> the direction here first would help me get it right before I start. >> >> Thanks, >> Amrit >> >
