[
https://issues.apache.org/jira/browse/QPID-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662121#comment-14662121
]
Keith Wall edited comment on QPID-6683 at 8/7/15 5:54 PM:
----------------------------------------------------------
The issue is that Qpid is constraining the Jetty *Server's* thread pool to 50
threads, and this is too few when many Jetty connectors are in use.
Here are the details:
# SelectChannelConnector#_accepts is defaults to
Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4 acceptors. Qpid
does not overriding this value.
# SelectorManager (SelectorManager#doStart) creates x number of "Selector"
threads, where x is the SelectorManager#_selectSets. _selectSets itself is
defaulted from SelectChannelConnector#_accepts
# AbstractConnector#doStart creates x number of "Acceptor" threads, where x is
the SelectChannelConnector#_accepts.
So, in the falling case, on the 80 core machine, Jetty desires to create 2 * 20
acceptor + 2 * 20 selectors = 80 threads, but this exceeds the configured
default 50 (from context variable "jetty.threadPool.maxThreads"). Increasing
"jetty.threadPool.maxThreads" to 80 gets rid of the "insufficient threads
configured" but the spinning "Dispatched Idle.." message remains. Jetty needs
to be able schedule at least one other task (the Idle tick in
SelectorManager#dispatch line 708) into its pool. With 81 threads all *seems*
okay, but it is hard to be sure without a full inspection of the Jetty code (I
can't find any docs that help). It is probably best to add some wriggle room
just in case :)
It seems to me we should:
# Have a Jetty ThreadPool per Connector (rather than per Server)
# Size the ThreadPool with maximum and minimum thread pool sizes coming from
HttpPort attributes (backed by context vars). Default sizes 5 and 50
respectively.
# Size SelectChannelConnector#_accepts to be (maximumThreadPool - wriggle room)
/ 2. Let wriggle room be say, 10. This value should come from a context var
(but not an attribute). Make sure maximumThreadPool - wriggle room is never
less than 2.
# Context variables on HttpManagementConfiguration will be removed. This is an
edge case, so I propose no upgrader change.
Comments welcome.
was (Author: k-wall):
The issue is that Qpid is constraining the Jetty *Server's* thread pool to 50
threads, and this is too few when many Jetty connectors are in use.
Here are the details:
# SelectChannelConnector#_accepts is defaults to
Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4 acceptors. Qpid
does not overriding this value.
# SelectorManager (SelectorManager#doStart) creates x number of "Selector"
threads, where x is the SelectorManager#_selectSets. _selectSets itself is
defaulted from SelectChannelConnector#_accepts
# AbstractConnector#doStart creates x number of "Acceptor" threads, where x is
the SelectChannelConnector#_accepts.
So, in the falling case, on the 80 core machine, Jetty desires to create 2 * 20
acceptor + 2 * 20 selectors = 80 threads, but this exceeds the configured
default 50 (from context variable "jetty.threadPool.maxThreads"). Increasing
"jetty.threadPool.maxThreads" to 80 gets rid of the "insufficient threads
configured" but the spinning "Dispatched Idle.." message remains. Jetty needs
to be able schedule at least one other task (the Idle tick in
SelectorManager#dispatch line 708) into its pool. With 81 threads all *seems*
okay, but it is hard to be sure without a full inspection of the Jetty code (I
can't find any docs that help). It is probably best to add some wriggle room
just in case :)
It seems to me we should:
# Have a Jetty ThreadPool per Connector (rather than per Server)
# Size the ThreadPool with maximum and minimum thread pool sizes coming from
HttpPort attributes (backed by context vars). Default sizes 5 and 50
respectively.
# Size SelectChannelConnector#_accepts to be (maximumThreadPool - wriggle room)
/ 2. Let wriggle room be say, 10. This value should come from a context var
(but not an attribute).
# Context variables on HttpManagementConfiguration will be removed. This is an
edge case, so I propose no upgrader change.
Comments welcome.
> Jetty thread seemingly spins if many HTTP ports are defined on hosts with
> large numbers of core
> -----------------------------------------------------------------------------------------------
>
> Key: QPID-6683
> URL: https://issues.apache.org/jira/browse/QPID-6683
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.32
> Reporter: Keith Wall
> Assignee: Keith Wall
> Fix For: qpid-java-6.0
>
>
> If I configure many HTTP management ports on a box with a large number of
> cores (80), I find that one of the two ports will be non-functional (produce
> a timeout in the browser/blank page). Turning on Jetty logs reveals a Jetty
> WARN during startup, during the attempt to configure the second port:
> {noformat}
> 2015-08-07 10:30:21,963 WARN [Broker-Config] (o.e.j.s.AbstractConnector) -
> insufficient threads configured for @0.0.0.0:8080
> {noformat}
> then evidence of a spinning Jetty thread once the connect
> {noformat}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-226 Selector2]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@37ca01b0 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-233 Selector9]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@2f34e66d to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-228 Selector4]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@3a041651 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-229 Selector5]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@3a041651 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,738 DEBUG [HttpManagement-230 Selector6]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@62553d98 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,738 DEBUG [HttpManagement-225 Selector1]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@4a3ef897 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,740 DEBUG [HttpManagement-232 Selector8]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@6c32a795 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,742 DEBUG [HttpManagement-227 Selector3]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@71985eb1 to stopped
> HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,745 DEBUG [HttpManagement-231 Selector7]
> (o.e.j.u.t.QueuedThreadPool) - Dispatched
> Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@4a70f0ba to stopped
> HttpManagement{5<=0<=50/50,1000}
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
