[
https://issues.apache.org/jira/browse/KNOX-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Csoma updated KNOX-1139:
-------------------------------
Description:
Currently Jetty calculates acceptor and selector count as a fraction of
processor count (core/8 and core/2 respectively) and will maximize them as 4.
That means 4 acceptor and 4 select in a 32+ core machine for each
{{ServerConnector}}. In {{Server.doStart}} jetty will check if
{{maxThreadPoolSize}} is large enough to store a request thread and all
acceptors and selectors thread for each connector together.
Currently in tests thread pool max size fixed as 16. In {{GatewayServer.start}}
if gateway port mapping is enabled it will initialize two connectors in jetty
which on a 32+ core machine will create 4+4 acceptor and 4+4 selector. With the
one extra request thread it will exceed the 16 thread pool limit and
{{org.apache.hadoop.gateway.GatewayPortMappingFuncTest} will throw the
following exception:
{code}
java.lang.IllegalStateException: Insufficient threads: max=16 <
needed(acceptors=8 + selectors=8 + request=1)
at
org.apache.hadoop.gateway.GatewayPortMappingFuncTest.setup(GatewayPortMappingFuncTest.java:96)
{code}
This is only a unit test problem since the default value for thread pool size
in {{GatewayConfigImpl}} is 254.
Probably it would be a better method to calculate thread pool count from core
count or max thread count in tests or just use the default value from
{{GatewayConfigImpl}}.
was:
Currently Jetty calculates acceptor and selector count as a fraction of
processor count (core/8 and core/2 respectively) and will maximize them as 4.
That means 4 acceptor and 4 select in a 32+ core machine for each
{{ServerConnector}}. In {{Server.doStart}} jetty will check if
{{maxThreadPoolSize}} is large enough to store a request thread and all
acceptors and selectors thread for each connector together.
Currently in tests thread pool max size fixed as 16. In {{GatewayServer.start}}
if gateway port mapping is enabled it will initialize two connectors in jetty
which on a 32+ core machine will create 4+4 acceptor and 4+4 selector. With the
one extra request thread it will exceed the 16 thread pool limit and
{{org.apache.hadoop.gateway.GatewayPortMappingFuncTest} will throw the
following exception:
{code}
java.lang.IllegalStateException: Insufficient threads: max=16 <
needed(acceptors=8 + selectors=8 + request=1)
at
org.apache.hadoop.gateway.GatewayPortMappingFuncTest.setup(GatewayPortMappingFuncTest.java:96)
{code}
This is only a unit test problem since the default value for thread pool size
in {{GatewayConfigImpl}} is 254.
Probably it would be a better method to calculate thread pool count from core
count in tests or just use the default value from {{GatewayConfigImpl}}.
> Tthread pool size in tests are not enough in a 32 core machine if gateway
> port mapping enabled
> -----------------------------------------------------------------------------------------------
>
> Key: KNOX-1139
> URL: https://issues.apache.org/jira/browse/KNOX-1139
> Project: Apache Knox
> Issue Type: Test
> Reporter: Attila Csoma
>
> Currently Jetty calculates acceptor and selector count as a fraction of
> processor count (core/8 and core/2 respectively) and will maximize them as 4.
> That means 4 acceptor and 4 select in a 32+ core machine for each
> {{ServerConnector}}. In {{Server.doStart}} jetty will check if
> {{maxThreadPoolSize}} is large enough to store a request thread and all
> acceptors and selectors thread for each connector together.
> Currently in tests thread pool max size fixed as 16. In
> {{GatewayServer.start}} if gateway port mapping is enabled it will initialize
> two connectors in jetty which on a 32+ core machine will create 4+4 acceptor
> and 4+4 selector. With the one extra request thread it will exceed the 16
> thread pool limit and {{org.apache.hadoop.gateway.GatewayPortMappingFuncTest}
> will throw the following exception:
> {code}
> java.lang.IllegalStateException: Insufficient threads: max=16 <
> needed(acceptors=8 + selectors=8 + request=1)
> at
> org.apache.hadoop.gateway.GatewayPortMappingFuncTest.setup(GatewayPortMappingFuncTest.java:96)
> {code}
> This is only a unit test problem since the default value for thread pool size
> in {{GatewayConfigImpl}} is 254.
> Probably it would be a better method to calculate thread pool count from core
> count or max thread count in tests or just use the default value from
> {{GatewayConfigImpl}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
