Hi,
I have a problem with RecipientList pattern. I'm using it in a parallel
processing mode which means it is backed up by a thread pool. The
configuration is as follows:
<threadPool id="pool" threadName="My RecipientList" poolSize="10"
maxPoolSize="200" />
<recipientList id="bidderResponseGatherer"
strategyRef="gatherBidderResponses" parallelProcessing="true"
stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true"
executorServiceRef="pool" >
<header>RecipientsList</header>
</recipientList>
The problem is that pool threads aren't released properly sometimes. After
a few hours following the start of an application I can see using jstack
tool that some threads are stuck in a WAITING state:
"Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10
tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fb4dc319940> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)
at
org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153)
at
org.apache.camel.processor.RecipientList.process(RecipientList.java:112)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136)
at
org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108)
at
org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199)
at
org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at
org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Number of such threads grows gradually until the application isn't able to
respond to the requests. I've found very similar ticket in your Jira (
https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there
that the problem is in the 3rd party library mqtt-client. However, I'm not
using this library, I'm only using Netty to make http requests but still
see a very similar behaviour.
I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as
suggested here
http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html.
But this didn't help too. Also I've noticed that thread is becoming
unavailable after the timeout specified for the calls has been reached.
Although not every timeout leads to a thread leak.
Can you suggest what else can be checked? Am I missing something? Or may be
this is a bug in Camel?
--
Alexander Ilyin
*tel: 856-874-8928 <856-874-8928>* Director of Engineering
[email protected] 717 Fellowship Rd. Suite D
Mount Laurel, NJ 08054
