[
https://issues.apache.org/jira/browse/BROOKLYN-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959581#comment-14959581
]
Aled Sage edited comment on BROOKLYN-181 at 10/15/15 8:50 PM:
--------------------------------------------------------------
Investigating further where all of these tasks are coming from:
{noformat}
Thread [brooklyn-jetty-server-8081-qtp9885135-1084] (Suspended (breakpoint at
line 549 in BasicExecutionManager))
owns: DynamicSequentialTask<T> (id=1868)
BasicExecutionManager.submitNewTask(Map<?,?>, Task<T>) line: 549
BasicExecutionManager.submit(Map<?,?>, TaskAdaptable<T>) line: 349
BasicExecutionContext.submitInternal(Map<?,?>, Object) line: 197
BasicExecutionContext(AbstractExecutionContext).submit(TaskAdaptable<T>) line:
57
ValueResolver<T>.getMaybeInternal() line: 330
ValueResolver<T>.getMaybe() line: 245
ValueResolver<T>.get() line: 238
AbstractBrooklynRestResource$RestValueResolver.getImmediateValue(Object,
Entity) line: 140
AbstractBrooklynRestResource$RestValueResolver.resolve() line: 129
EntityConfigResource.batchConfigRead(String, String, Boolean) line: 80
GeneratedMethodAccessor50.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 606
JavaMethodInvokerFactory$1.invoke(Method, Object, Object...) line: 60
AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(Object,
HttpContext) line: 185
AbstractResourceMethodDispatchProvider$TypeOutInvoker(ResourceJavaMethodDispatcher).dispatch(Object,
HttpContext) line: 75
HttpMethodRule.accept(CharSequence, Object, UriRuleContext) line: 302
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
ResourceObjectRule.accept(CharSequence, Object, UriRuleContext) line:
100
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
RootResourceClassesRule.accept(CharSequence, Object, UriRuleContext)
line: 84
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest) line: 1542
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest, ContainerResponse) line: 1473
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponse)
line: 1419
WebApplicationImpl.handleRequest(ContainerRequest,
ContainerResponseWriter) line: 1409
ServletContainer$InternalWebComponent(WebComponent).service(URI, URI,
HttpServletRequest, HttpServletResponse) line: 409
ServletContainer.service(URI, URI, HttpServletRequest,
HttpServletResponse) line: 540
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain, String, String, String) line: 909
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain) line: 857
ServletContainer.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 811
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
HaMasterCheckFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 101
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 89
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
BrooklynPropertiesSecurityFilter.doFilter(ServletRequest,
ServletResponse, FilterChain) line: 122
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
RequestTaggingFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 49
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
ServletHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 501
ServletHandler(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 137
ConstraintSecurityHandler(SecurityHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 557
SessionHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 231
WebAppContext(ContextHandler).doHandle(String, Request,
HttpServletRequest, HttpServletResponse) line: 1086
ServletHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 429
SessionHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 193
WebAppContext(ContextHandler).doScope(String, Request,
HttpServletRequest, HttpServletResponse) line: 1020
WebAppContext(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 135
ContextHandlerCollectionHotSwappable(ContextHandlerCollection).handle(String,
Request, HttpServletRequest, HttpServletResponse) line: 255
Server(HandlerWrapper).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 116
Server.handle(AbstractHttpConnection) line: 370
AsyncHttpConnection(AbstractHttpConnection).handleRequest() line: 494
AsyncHttpConnection(AbstractHttpConnection).headerComplete() line: 971
AbstractHttpConnection$RequestHandler.headerComplete() line: 1033
HttpParser.parseNext() line: 644
HttpParser.parseAvailable() line: 235
AsyncHttpConnection.handle() line: 82
SelectChannelEndPoint.handle() line: 696
SelectChannelEndPoint$1.run() line: 53
QueuedThreadPool.runJob(Runnable) line: 608
QueuedThreadPool$3.run() line: 543
Thread.run() line: 745
{noformat}
In other words, it caused by the polling of the web-console. The rest api's
{{AbstractBrooklynRestResource$RestValueResolver.resolve()}} is doing:
{noformat}
return
Tasks.resolving(value).as(Object.class).defaultValue(UNRESOLVED).timeout(Duration.ZERO).context(context).swallowExceptions().get();
{noformat}
which is supposed to execute immediately without blocking. But there is another
task that is blocked on this (without a timeout), and is in the synchronized
block (holding that lock) in
{{org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get}}.
I think the reason the riak entity gets into this state is that the tomcat
server is configured with:
{noformat}
java.sysprops:
brooklyn.example.riak.nodes:
$brooklyn:component("cluster").attributeWhenReady("riak.cluster.nodeList")
{noformat}
When it tries to do {{tomcat.stop()}}, it tries to populate the environment
variables to then execute the stop script. This involves getting the sysprops.
However, it waits for "riak.cluster.nodeList" (indefinitely because that sensor
is now an empty list again). Therefore the stop hangs, and it keeps holding the
lock. Indeed this is confirmed by looking at jstack, which shows one of the
threads doing:
{noformat}
"brooklyn-execmanager-eIDvgu20-164" daemon prio=5 tid=0x00007fdbde8d2800
nid=0x12b23 waiting on condition [0x00007000068f5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e4726b18> (a
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:342)
at org.apache.brooklyn.util.time.Durations.get(Durations.java:43)
at org.apache.brooklyn.util.time.Durations.get(Durations.java:64)
at
org.apache.brooklyn.util.core.task.ValueResolver.getMaybeInternal(ValueResolver.java:336)
at
org.apache.brooklyn.util.core.task.ValueResolver.getMaybe(ValueResolver.java:245)
at
org.apache.brooklyn.util.core.task.ValueResolver.get(ValueResolver.java:238)
at org.apache.brooklyn.util.core.task.Tasks.resolveValue(Tasks.java:139)
at
org.apache.brooklyn.core.config.BasicConfigKey.resolveValue(BasicConfigKey.java:248)
at
org.apache.brooklyn.core.config.BasicConfigKey.extractValue(BasicConfigKey.java:233)
at
org.apache.brooklyn.core.config.SubElementConfigKey.extractValue(SubElementConfigKey.java:48)
at
org.apache.brooklyn.core.config.internal.AbstractStructuredConfigKey.extractValue(AbstractStructuredConfigKey.java:111)
at
org.apache.brooklyn.core.config.internal.AbstractStructuredConfigKey.extractValue(AbstractStructuredConfigKey.java:124)
at
org.apache.brooklyn.core.entity.internal.EntityConfigMap.getConfig(EntityConfigMap.java:116)
at
org.apache.brooklyn.core.config.internal.AbstractConfigMapImpl.getConfig(AbstractConfigMapImpl.java:50)
at
org.apache.brooklyn.core.entity.AbstractEntity$BasicConfigurationSupport.get(AbstractEntity.java:1141)
at
org.apache.brooklyn.core.entity.AbstractEntity.getConfig(AbstractEntity.java:1247)
at
org.apache.brooklyn.entity.java.JavaSoftwareProcessSshDriver.getJavaSystemProperties(JavaSoftwareProcessSshDriver.java:208)
at
org.apache.brooklyn.entity.java.JavaSoftwareProcessSshDriver.getJavaOpts(JavaSoftwareProcessSshDriver.java:125)
at
org.apache.brooklyn.entity.java.JavaSoftwareProcessSshDriver.getShellEnvironment(JavaSoftwareProcessSshDriver.java:105)
at
org.apache.brooklyn.entity.webapp.tomcat.TomcatSshDriver.getShellEnvironment(TomcatSshDriver.java:140)
at
org.apache.brooklyn.entity.software.base.AbstractSoftwareProcessSshDriver.execute(AbstractSoftwareProcessSshDriver.java:261)
at
org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper.executeInternal(ScriptHelper.java:366)
at
org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper$8.call(ScriptHelper.java:287)
at
org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper$8.call(ScriptHelper.java:1)
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
was (Author: aled.sage):
Investigating further where all of these tasks are coming from:
{noformat}
Thread [brooklyn-jetty-server-8081-qtp9885135-1084] (Suspended (breakpoint at
line 549 in BasicExecutionManager))
owns: DynamicSequentialTask<T> (id=1868)
BasicExecutionManager.submitNewTask(Map<?,?>, Task<T>) line: 549
BasicExecutionManager.submit(Map<?,?>, TaskAdaptable<T>) line: 349
BasicExecutionContext.submitInternal(Map<?,?>, Object) line: 197
BasicExecutionContext(AbstractExecutionContext).submit(TaskAdaptable<T>) line:
57
ValueResolver<T>.getMaybeInternal() line: 330
ValueResolver<T>.getMaybe() line: 245
ValueResolver<T>.get() line: 238
AbstractBrooklynRestResource$RestValueResolver.getImmediateValue(Object,
Entity) line: 140
AbstractBrooklynRestResource$RestValueResolver.resolve() line: 129
EntityConfigResource.batchConfigRead(String, String, Boolean) line: 80
GeneratedMethodAccessor50.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 606
JavaMethodInvokerFactory$1.invoke(Method, Object, Object...) line: 60
AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(Object,
HttpContext) line: 185
AbstractResourceMethodDispatchProvider$TypeOutInvoker(ResourceJavaMethodDispatcher).dispatch(Object,
HttpContext) line: 75
HttpMethodRule.accept(CharSequence, Object, UriRuleContext) line: 302
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
ResourceObjectRule.accept(CharSequence, Object, UriRuleContext) line:
100
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
RootResourceClassesRule.accept(CharSequence, Object, UriRuleContext)
line: 84
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest) line: 1542
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest, ContainerResponse) line: 1473
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponse)
line: 1419
WebApplicationImpl.handleRequest(ContainerRequest,
ContainerResponseWriter) line: 1409
ServletContainer$InternalWebComponent(WebComponent).service(URI, URI,
HttpServletRequest, HttpServletResponse) line: 409
ServletContainer.service(URI, URI, HttpServletRequest,
HttpServletResponse) line: 540
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain, String, String, String) line: 909
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain) line: 857
ServletContainer.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 811
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
HaMasterCheckFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 101
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 89
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
BrooklynPropertiesSecurityFilter.doFilter(ServletRequest,
ServletResponse, FilterChain) line: 122
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
RequestTaggingFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 49
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
ServletHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 501
ServletHandler(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 137
ConstraintSecurityHandler(SecurityHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 557
SessionHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 231
WebAppContext(ContextHandler).doHandle(String, Request,
HttpServletRequest, HttpServletResponse) line: 1086
ServletHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 429
SessionHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 193
WebAppContext(ContextHandler).doScope(String, Request,
HttpServletRequest, HttpServletResponse) line: 1020
WebAppContext(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 135
ContextHandlerCollectionHotSwappable(ContextHandlerCollection).handle(String,
Request, HttpServletRequest, HttpServletResponse) line: 255
Server(HandlerWrapper).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 116
Server.handle(AbstractHttpConnection) line: 370
AsyncHttpConnection(AbstractHttpConnection).handleRequest() line: 494
AsyncHttpConnection(AbstractHttpConnection).headerComplete() line: 971
AbstractHttpConnection$RequestHandler.headerComplete() line: 1033
HttpParser.parseNext() line: 644
HttpParser.parseAvailable() line: 235
AsyncHttpConnection.handle() line: 82
SelectChannelEndPoint.handle() line: 696
SelectChannelEndPoint$1.run() line: 53
QueuedThreadPool.runJob(Runnable) line: 608
QueuedThreadPool$3.run() line: 543
Thread.run() line: 745
{noformat}
In other words, it caused by the polling of the web-console. The rest api's
{{AbstractBrooklynRestResource$RestValueResolver.resolve()}} is doing:
{noformat}
return
Tasks.resolving(value).as(Object.class).defaultValue(UNRESOLVED).timeout(Duration.ZERO).context(context).swallowExceptions().get();
{noformat}
which is supposed to execute immediately without blocking. But there is another
task that is blocked on this (without a timeout), and is in the synchronized
block (holding that lock) in
{{org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get}}.
> OutOfMemoryError: unable to create new native thread (when stopping app)
> ------------------------------------------------------------------------
>
> Key: BROOKLYN-181
> URL: https://issues.apache.org/jira/browse/BROOKLYN-181
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Aled Sage
> Attachments: many-threads.txt
>
>
> Using 0.9.0-SNAPSHOT...
> I deployed an app to OpenStack (Bluebox) - it failed due to insufficient
> security groups etc. I terminated the VMs in Bluebox itself, and then I tried
> to call the "Stop" effector on the app in Brooklyn. Eventually Brooklyn ran
> out of threads.
> My app consisted of a riak cluster + web cluster.
> However, the log is messy because I had a couple of other apps that I also
> started/stopped during this time (I think they are unrelated though).
> There are 2045 threads, of which 2012 are of the form
> "brooklyn-execmanager-*".
> 998 of these threads are doing:
> {noformat}
> "brooklyn-execmanager-vvDaziiT-2847" daemon prio=5 tid=0x00007fd4f08aa800
> nid=0xc7d0f waiting for monitor entry [0x0000700063d3f000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:83)
> - waiting to lock <0x00000007c5940bc8> (a
> org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
> at
> org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Another 999 threads look like:
> {noformat}
> "brooklyn-execmanager-vvDaziiT-2930" daemon prio=5 tid=0x00007fd4ef16f800
> nid=0x5e07 in Object.wait() [0x0000700001703000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:275)
> - locked <0x00000007c64bafa8> (a java.lang.Object)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:265)
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The logging shows the number of threads creeping up, with 30 more threads and
> 30 more active tasks being added every minute, e.g.:
> {noformat}
> 2015-10-12 23:01:14,022 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector
> [brooklyn-gc]: brooklyn gc (after) - using 164 MB / 264 MB memory (20.9 kB
> soft); 2001 threads; storage: {datagrid={size=7, createCount=7},
> refsMapSize=0, lis
> tsMapSize=0}; tasks: 1981 active, 999 unfinished; 1084 remembered, 286554
> total submitted)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
