yanbin.zhang created HADOOP-20054:
-------------------------------------
Summary: Expose Jetty QueuedThreadPool metrics in
HttpServer2Metrics
Key: HADOOP-20054
URL: https://issues.apache.org/jira/browse/HADOOP-20054
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Affects Versions: 3.5.0
Reporter: yanbin.zhang
HADOOP-17133 added HttpServer2Metrics, which exposes Jetty's
StatisticsHandler (request counts, response codes, request/dispatch
latency, etc.) as Hadoop Metrics2.
However, it does not expose anything about the Jetty HTTP worker thread
pool (QueuedThreadPool). When a daemon's web endpoint (e.g. the RM at
:8088, or NN/DN/NM web UIs) becomes slow or unresponsive, there is
currently no metric to tell whether the HTTP thread pool is exhausted or
whether requests are piling up in the queue. Operators can only guess, or
take a jstack and count "qtp*" threads manually after the fact.
A common failure mode is that web worker threads block (for example while
acquiring a scheduler lock), the pool saturates at its max (250 by
default), new requests queue up, clients time out and drop the connection
-- surfacing as jetty EofException on the server side -- while the daemon
process itself is otherwise healthy. Without thread-pool metrics this is
hard to confirm.
This improvement extends HttpServer2Metrics to also expose the
QueuedThreadPool state:
- HttpThreadsMax (getMaxThreads)
- HttpThreadsMin (getMinThreads)
- HttpThreads (getThreads)
- HttpThreadsIdle (getIdleThreads)
- HttpThreadsBusy (getBusyThreads)
- HttpThreadsQueueSize (getQueueSize)
- HttpThreadsLowOnThreads(isLowOnThreads, as 0/1)
These are published under the existing "HttpServer2-<port>" metrics
source and remain gated by the existing hadoop.http.metrics.enabled
switch, so there is no behavior change when metrics are disabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
