[
https://issues.apache.org/jira/browse/SOLR-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137210#comment-18137210
]
Jan Høydahl commented on SOLR-18616:
------------------------------------
Thanks for bringing this up Hoss. A clear oversight which shuold be plugged. I
burned some Fable tokens, and it could not either find anything obvious apart
from your suggestion in the last paragraph. Here is verbatim 🤖 options list:
# *Pull-bridge with Jetty's own StatisticsHandler (recommended first step).*
{{jetty.xml}} inserts a {{StatisticsHandler}} at the top of the chain plus
{{ConnectionStatistics}} on the connectors, both already shipped in
{{{}server/lib{}}}. At context init, solr-core locates the handler and the
{{QueuedThreadPool}} and wraps their getters as observable gauges and counters
in a new {{solr.jetty}} registry. The metrics handler and OTLP export pick up
the registry automatically. No new dependency. Gives requests total and active,
responses by 1xx to 5xx class, failures, bytes, connections, and full
thread-pool utilization. Gap: no per-HTTP-method breakdown and status classes
only.
# *Custom EventsHandler for per-method and exact-status counts.* Covers all
traffic including non-Solr paths. Needs a class on the server classloader, so
either a new tiny Gradle module packaged into {{{}server/lib/ext{}}}, or a
programmatic insert from the webapp during startup that needs a spike to
confirm Jetty allows it.
# *Servlet-layer filter in the webapp.* Either the OpenTelemetry
{{opentelemetry-servlet-5.0}} library filter fed through the same
fake-OpenTelemetry trick used for JVM metrics, or a small home-grown filter.
Gives exact status codes and methods with standard names, but never sees
requests rejected before the webapp and gives no thread-pool data.
# *Jetty JMX plus the OpenTelemetry jmx-metrics library.* {{jetty-jmx}} is
already in {{server/lib}} but unconfigured. Indirect, another alpha dependency,
and its bundled rules cover threads but not HTTP responses. Not worth it over
option 1.
# *OpenTelemetry Java agent as an operator workaround.* Emits standard HTTP
server metrics but into its own pipeline, not into the metrics endpoint.
Documentation note only.
# *At minimum,* add an upgrade note that the {{solr.jetty}} registry is gone,
since neither the 10.1 changelog nor the ref guide mentions it.
Recommendation: Do option 1 now, since it closes both headline gaps with no new
dependencies. Then decide separately whether per-method and exact-status detail
justifies option 2 or 3.
> Solr 10.+ no longer provides any jetty metrics
> ----------------------------------------------
>
> Key: SOLR-18616
> URL: https://issues.apache.org/jira/browse/SOLR-18616
> Project: Solr
> Issue Type: Sub-task
> Reporter: Chris M. Hostetter
> Priority: Major
>
> [https://lists.apache.org/thread.html/8yzggfvrohv09pqs2wgq1y1nfsq80xhc]
>
> Prior to Solr 10, Solr leveraged 2 "dropwizard" provided jetty plugins in
> order to report some key jetty metrics...
> * InstrumentedEE10Handler
> * InstrumentedQueuedThreadPool
> Those were removed by SOLR-17855, w/o any replacement:
> [https://github.com/apache/solr/commit/cee53df8e0a]
>
> The major impacts of this loss are:
> # no other (existing) solr metrics provide visibility into the number of
> HTTP requests, and their associated responses that do *NOT* a valid Solr
> collection/core
> ** so for example: there is no indication in the existing 10.x metrics of a
> huge spike of abnormal {{PUT}} requests to a patch that doesn't exist.
> # no other (existing) solr metrics provide visibility into the thread
> utilization of the jetty (request) thread pool
> ** so for example: there is no indication in the existing 10.x metrics that
> the current number of currently executing requests is growing continuously
> and getting close to saturating the max number of threads
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
