Amrit Sarkar created SOLR-18571:
-----------------------------------
Summary: Query Insights: opt-in top-N expensive queries per node
Key: SOLR-18571
URL: https://issues.apache.org/jira/browse/SOLR-18571
Project: Solr
Issue Type: New Feature
Components: search, v2 API, metrics
Reporter: Amrit Sarkar
Umbrella for Query Insights: an opt-in module that keeps the most expensive
searches on each node in memory and returns them through one v2 API across the
cluster. It implements the idea in SOLR-11994.
Discussed on dev@:
[thread|https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd]. No
objections came in on the thread.
*Shape*
* Off by default. Turned on by adding {{query-insights}} to {{SOLR_MODULES}}
and setting {{SOLR_QUERYINSIGHTS_ENABLED=true}}. Without both, nothing loads
and each search pays one {{isEmpty()}} check.
* Records only on the coordinator. Top-N by latency and by CPU over
clock-aligned windows; the current and previous window are returned. Defaults:
N=10 (max 100), 5 minute windows.
* Each record has collection, handler, params (redactable with
{{params.hidden}}, capped at 8 KB), hits, shard count, user, trace id, latency,
CPU, and the costly features the query used (re-ranking, LTR, knn, grouping,
faceting, paramsets).
* {{GET /api/node/query-insights/top-queries?type=latency|cpu&nodes=all}},
needs {{metrics-read}}. Optional {{collection}} and {{traceId}} filters. Nodes
that don't answer are listed in {{failedNodes}}.
* Defaults from system properties; runtime changes through
{{ext.queryInsights.*}} cluster properties.
*Plan* (one PR each, each reviewable and mergeable on its own)
# SOLR-11994: {{SearchRequestListener}} hook in core. No behavior change when
nothing is registered.
# The {{query-insights}} module: per-node top-N and the node-local API.
# Cluster-wide view: {{nodes=all}} merge with {{failedNodes}}.
# {{collection}} and {{traceId}} filters, runtime settings through cluster
properties.
# Ref guide page.
*Defaults for the open questions from the thread* (happy to change any of these
in review)
* Module plus a small core listener, rather than all in core.
* {{metrics-read}} permission, with {{params.hidden}} redaction.
* {{SearchRequestListener}} marked {{@lucene.experimental}}.
* {{/export}} is recorded; {{/get}} is not.
*Later, separate sub-tasks*: stored history as a Time Routed Alias with
{{autoDeleteAge}} (to be lined up with [UBI PR
#2452|https://github.com/apache/solr/pull/2452]), grouping similar queries by
shape, memory as a metric, an Admin UI page.
Full design is attached as {{query-insights-design.pdf}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
