[
https://issues.apache.org/jira/browse/BROOKLYN-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959581#comment-14959581
]
Aled Sage commented on BROOKLYN-181:
------------------------------------
Investigating further where all of these tasks are coming from:
{noformat}
Thread [brooklyn-jetty-server-8081-qtp9885135-1084] (Suspended (breakpoint at
line 549 in BasicExecutionManager))
owns: DynamicSequentialTask<T> (id=1868)
BasicExecutionManager.submitNewTask(Map<?,?>, Task<T>) line: 549
BasicExecutionManager.submit(Map<?,?>, TaskAdaptable<T>) line: 349
BasicExecutionContext.submitInternal(Map<?,?>, Object) line: 197
BasicExecutionContext(AbstractExecutionContext).submit(TaskAdaptable<T>) line:
57
ValueResolver<T>.getMaybeInternal() line: 330
ValueResolver<T>.getMaybe() line: 245
ValueResolver<T>.get() line: 238
AbstractBrooklynRestResource$RestValueResolver.getImmediateValue(Object,
Entity) line: 140
AbstractBrooklynRestResource$RestValueResolver.resolve() line: 129
EntityConfigResource.batchConfigRead(String, String, Boolean) line: 80
GeneratedMethodAccessor50.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 606
JavaMethodInvokerFactory$1.invoke(Method, Object, Object...) line: 60
AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(Object,
HttpContext) line: 185
AbstractResourceMethodDispatchProvider$TypeOutInvoker(ResourceJavaMethodDispatcher).dispatch(Object,
HttpContext) line: 75
HttpMethodRule.accept(CharSequence, Object, UriRuleContext) line: 302
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
ResourceObjectRule.accept(CharSequence, Object, UriRuleContext) line:
100
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line:
147
RootResourceClassesRule.accept(CharSequence, Object, UriRuleContext)
line: 84
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest) line: 1542
WebApplicationImpl._handleRequest(WebApplicationContext,
ContainerRequest, ContainerResponse) line: 1473
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponse)
line: 1419
WebApplicationImpl.handleRequest(ContainerRequest,
ContainerResponseWriter) line: 1409
ServletContainer$InternalWebComponent(WebComponent).service(URI, URI,
HttpServletRequest, HttpServletResponse) line: 409
ServletContainer.service(URI, URI, HttpServletRequest,
HttpServletResponse) line: 540
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain, String, String, String) line: 909
ServletContainer.doFilter(HttpServletRequest, HttpServletResponse,
FilterChain) line: 857
ServletContainer.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 811
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
HaMasterCheckFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 101
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain)
line: 89
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
BrooklynPropertiesSecurityFilter.doFilter(ServletRequest,
ServletResponse, FilterChain) line: 122
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
RequestTaggingFilter.doFilter(ServletRequest, ServletResponse,
FilterChain) line: 49
ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse)
line: 1467
ServletHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 501
ServletHandler(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 137
ConstraintSecurityHandler(SecurityHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 557
SessionHandler.doHandle(String, Request, HttpServletRequest,
HttpServletResponse) line: 231
WebAppContext(ContextHandler).doHandle(String, Request,
HttpServletRequest, HttpServletResponse) line: 1086
ServletHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 429
SessionHandler.doScope(String, Request, HttpServletRequest,
HttpServletResponse) line: 193
WebAppContext(ContextHandler).doScope(String, Request,
HttpServletRequest, HttpServletResponse) line: 1020
WebAppContext(ScopedHandler).handle(String, Request,
HttpServletRequest, HttpServletResponse) line: 135
ContextHandlerCollectionHotSwappable(ContextHandlerCollection).handle(String,
Request, HttpServletRequest, HttpServletResponse) line: 255
Server(HandlerWrapper).handle(String, Request, HttpServletRequest,
HttpServletResponse) line: 116
Server.handle(AbstractHttpConnection) line: 370
AsyncHttpConnection(AbstractHttpConnection).handleRequest() line: 494
AsyncHttpConnection(AbstractHttpConnection).headerComplete() line: 971
AbstractHttpConnection$RequestHandler.headerComplete() line: 1033
HttpParser.parseNext() line: 644
HttpParser.parseAvailable() line: 235
AsyncHttpConnection.handle() line: 82
SelectChannelEndPoint.handle() line: 696
SelectChannelEndPoint$1.run() line: 53
QueuedThreadPool.runJob(Runnable) line: 608
QueuedThreadPool$3.run() line: 543
Thread.run() line: 745
{noformat}
In other words, it caused by the polling of the web-console. The rest api's
{{AbstractBrooklynRestResource$RestValueResolver.resolve()}} is doing:
{noformat}
return
Tasks.resolving(value).as(Object.class).defaultValue(UNRESOLVED).timeout(Duration.ZERO).context(context).swallowExceptions().get();
{noformat}
which is supposed to execute immediately without blocking. But there is another
task that is blocked on this (without a timeout), and is in the synchronized
block (holding that lock) in
{{org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get}}.
> OutOfMemoryError: unable to create new native thread (when stopping app)
> ------------------------------------------------------------------------
>
> Key: BROOKLYN-181
> URL: https://issues.apache.org/jira/browse/BROOKLYN-181
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Aled Sage
> Attachments: many-threads.txt
>
>
> Using 0.9.0-SNAPSHOT...
> I deployed an app to OpenStack (Bluebox) - it failed due to insufficient
> security groups etc. I terminated the VMs in Bluebox itself, and then I tried
> to call the "Stop" effector on the app in Brooklyn. Eventually Brooklyn ran
> out of threads.
> My app consisted of a riak cluster + web cluster.
> However, the log is messy because I had a couple of other apps that I also
> started/stopped during this time (I think they are unrelated though).
> There are 2045 threads, of which 2012 are of the form
> "brooklyn-execmanager-*".
> 998 of these threads are doing:
> {noformat}
> "brooklyn-execmanager-vvDaziiT-2847" daemon prio=5 tid=0x00007fd4f08aa800
> nid=0xc7d0f waiting for monitor entry [0x0000700063d3f000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:83)
> - waiting to lock <0x00000007c5940bc8> (a
> org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
> at
> org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Another 999 threads look like:
> {noformat}
> "brooklyn-execmanager-vvDaziiT-2930" daemon prio=5 tid=0x00007fd4ef16f800
> nid=0x5e07 in Object.wait() [0x0000700001703000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:275)
> - locked <0x00000007c64bafa8> (a java.lang.Object)
> at
> org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:265)
> at
> org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The logging shows the number of threads creeping up, with 30 more threads and
> 30 more active tasks being added every minute, e.g.:
> {noformat}
> 2015-10-12 23:01:14,022 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector
> [brooklyn-gc]: brooklyn gc (after) - using 164 MB / 264 MB memory (20.9 kB
> soft); 2001 threads; storage: {datagrid={size=7, createCount=7},
> refsMapSize=0, lis
> tsMapSize=0}; tasks: 1981 active, 999 unfinished; 1084 remembered, 286554
> total submitted)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
