[
https://issues.apache.org/jira/browse/HADOOP-20054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136380#comment-18136380
]
ASF GitHub Bot commented on HADOOP-20054:
-----------------------------------------
singer-bin opened a new pull request, #8909:
URL: https://github.com/apache/hadoop/pull/8909
### Description of PR
JIRA: [HADOOP-20054](https://issues.apache.org/jira/browse/HADOOP-20054)
HADOOP-17133 added `HttpServer2Metrics`, which exposes Jetty's
`StatisticsHandler` (request counts, response codes, request/dispatch
latency, etc.) as Hadoop Metrics2. However, it does not expose anything
about the Jetty HTTP worker thread pool (`QueuedThreadPool`).
When a daemon's web endpoint (e.g. the RM at `:8088`, or the NN/DN/NM web
UIs) becomes slow or unresponsive, there is currently no metric telling
whether the HTTP thread pool is exhausted or whether requests are piling
up in the queue. Operators can only guess, or take a `jstack` and count
`qtp*` threads manually after the fact.
A common failure mode: web worker threads block (for example while
acquiring a scheduler lock), the pool saturates at its max (250 by
default), new requests queue up, clients time out and drop the
connection
> Expose Jetty QueuedThreadPool metrics in HttpServer2Metrics
> -----------------------------------------------------------
>
> Key: HADOOP-20054
> URL: https://issues.apache.org/jira/browse/HADOOP-20054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
> Affects Versions: 3.5.0
> Reporter: yanbin.zhang
> Priority: Major
>
> HADOOP-17133 added HttpServer2Metrics, which exposes Jetty's
> StatisticsHandler (request counts, response codes, request/dispatch
> latency, etc.) as Hadoop Metrics2.
> However, it does not expose anything about the Jetty HTTP worker thread
> pool (QueuedThreadPool). When a daemon's web endpoint (e.g. the RM at
> :8088, or NN/DN/NM web UIs) becomes slow or unresponsive, there is
> currently no metric to tell whether the HTTP thread pool is exhausted or
> whether requests are piling up in the queue. Operators can only guess, or
> take a jstack and count "qtp*" threads manually after the fact.
> A common failure mode is that web worker threads block (for example while
> acquiring a scheduler lock), the pool saturates at its max (250 by
> default), new requests queue up, clients time out and drop the connection
> -- surfacing as jetty EofException on the server side -- while the daemon
> process itself is otherwise healthy. Without thread-pool metrics this is
> hard to confirm.
> This improvement extends HttpServer2Metrics to also expose the
> QueuedThreadPool state:
> - HttpThreadsMax (getMaxThreads)
> - HttpThreadsMin (getMinThreads)
> - HttpThreads (getThreads)
> - HttpThreadsIdle (getIdleThreads)
> - HttpThreadsBusy (getBusyThreads)
> - HttpThreadsQueueSize (getQueueSize)
> - HttpThreadsLowOnThreads(isLowOnThreads, as 0/1)
> These are published under the existing "HttpServer2-<port>" metrics
> source and remain gated by the existing hadoop.http.metrics.enabled
> switch, so there is no behavior change when metrics are disabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
