Hi Claus, thank you for your answer. I hadn't configured netty request timeout before I wrote my first letter. Now I've read that tasks may keep running when using RecipientList timeout and have set up netty request timeout instead of RecipientList timeout as you suggested. But the situation didn't change much. NettyOrderedWorker threads are still getting stuck in a WAITING state after some time following the start of an app as well as RecipientList threads. What else can be checked? I can provide additional info if needed.
On Mon, Apr 20, 2015 at 4:19 PM, Alexander Ilyin < [email protected]> wrote: > Hi, > > I have a problem with RecipientList pattern. I'm using it in a parallel > processing mode which means it is backed up by a thread pool. The > configuration is as follows: > > <threadPool id="pool" threadName="My RecipientList" poolSize="10" > maxPoolSize="200" /> > > <recipientList id="bidderResponseGatherer" > strategyRef="gatherBidderResponses" parallelProcessing="true" > stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true" > executorServiceRef="pool" > > <header>RecipientsList</header> > </recipientList> > > The problem is that pool threads aren't released properly sometimes. After > a few hours following the start of an application I can see using jstack > tool that some threads are stuck in a WAITING state: > > "Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10 > tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007fb4dc319940> (a > java.util.concurrent.CountDownLatch$Sync) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at > org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327) > at > org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213) > at > org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153) > at > org.apache.camel.processor.RecipientList.process(RecipientList.java:112) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at > org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136) > at > org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108) > at > org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199) > at > org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Number of such threads grows gradually until the application isn't able to > respond to the requests. I've found very similar ticket in your Jira ( > https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there > that the problem is in the 3rd party library mqtt-client. However, I'm not > using this library, I'm only using Netty to make http requests but still > see a very similar behaviour. > > I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as > suggested here > http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html. > But this didn't help too. Also I've noticed that thread is becoming > unavailable after the timeout specified for the calls has been reached. > Although not every timeout leads to a thread leak. > > Can you suggest what else can be checked? Am I missing something? Or may > be this is a bug in Camel? > > > -- > Alexander Ilyin > *tel: 856-874-8928 <856-874-8928>* Director of Engineering > [email protected] 717 Fellowship Rd. Suite D > Mount Laurel, NJ 08054 > -- Alexander Ilyin *tel: 856-874-8928 <856-874-8928>* Director of Engineering [email protected] 717 Fellowship Rd. Suite D Mount Laurel, NJ 08054
