Hi Have you configured netty to use request timeout? http://camel.apache.org/netty.html
You should favor using that for timeout, and not as much the timeout on the recipient list. Or make sure those values are so netty timeout first. On Mon, Apr 20, 2015 at 3:19 PM, Alexander Ilyin <alexande...@madadsmedia.com> wrote: > Hi, > > I have a problem with RecipientList pattern. I'm using it in a parallel > processing mode which means it is backed up by a thread pool. The > configuration is as follows: > > <threadPool id="pool" threadName="My RecipientList" poolSize="10" > maxPoolSize="200" /> > > <recipientList id="bidderResponseGatherer" > strategyRef="gatherBidderResponses" parallelProcessing="true" > stopOnException="false" timeout="1000" ignoreInvalidEndpoints="true" > executorServiceRef="pool" > > <header>RecipientsList</header> > </recipientList> > > The problem is that pool threads aren't released properly sometimes. After > a few hours following the start of an application I can see using jstack > tool that some threads are stuck in a WAITING state: > > "Camel (mainContext) thread #187456 - NettyOrderedWorker" daemon prio=10 > tid=0x00007fb14c058000 nid=0x5564 waiting on condition [0x00007fb2472d0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00007fb4dc319940> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at > java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at > org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:327) > at > org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213) > at > org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:153) > at > org.apache.camel.processor.RecipientList.process(RecipientList.java:112) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at > org.apache.camel.component.netty.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:136) > at > org.apache.camel.component.netty.handlers.ServerChannelHandler.messageReceived(ServerChannelHandler.java:108) > at > org.apache.camel.component.netty.http.handlers.HttpServerChannelHandler.messageReceived(HttpServerChannelHandler.java:199) > at > org.apache.camel.component.netty.http.handlers.HttpServerMultiplexChannelHandler.messageReceived(HttpServerMultiplexChannelHandler.java:103) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:314) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > Number of such threads grows gradually until the application isn't able to > respond to the requests. I've found very similar ticket in your Jira ( > https://issues.apache.org/jira/browse/CAMEL-6717) but it is stated there > that the problem is in the 3rd party library mqtt-client. However, I'm not > using this library, I'm only using Netty to make http requests but still > see a very similar behaviour. > > I've updated Apache Camel to the latest version 2.15.1 (from 2.12.2) as > suggested here > http://camel.465427.n5.nabble.com/Camel-Netty-ReadTimeout-event-not-trigerred-td5761676.html. > But this didn't help too. Also I've noticed that thread is becoming > unavailable after the timeout specified for the calls has been reached. > Although not every timeout leads to a thread leak. > > Can you suggest what else can be checked? Am I missing something? Or may be > this is a bug in Camel? > > > -- > Alexander Ilyin > *tel: 856-874-8928 <856-874-8928>* Director of Engineering > alexande...@madadsmedia.com 717 Fellowship Rd. Suite D > Mount Laurel, NJ 08054 -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
