Hi Also dont stop Camel from the same thread you process a message. You should spin that off to a separate thread, or use controlbus in async mode etc.
On Mon, Jul 13, 2015 at 11:18 AM, jnupponen <[email protected]> wrote: > Hi! > > I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2) > cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler. > If I want to shutdown/restart routes during DefaultErrorHandlers > redeliveries quartz scheduling gets deadlocked and quietly fails all future > fire times. Without AsyncDelayedDelivery-option this problem does not occur. > I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs > when running the code on jvm (at least on 1.7) without Karaf. > > I have configured quite long redelivery times for my routes (e.g. 8 times in > 2 hour interval) but at times I would need to be able to restart bundles > running in Karaf even when there would be redeliveries still left (e.g. when > installing new version of the bundle). > > This seems like a bug to me but what's your take on this? I've attached few > lines of log and some example route code to the end of this message. > > Best regards, > - Jussi > > > > Logs on org.quartz debug level show every few seconds: > [artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG > Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down > > and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2: > "DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000 > nid=0x2a16 in Object.wait() [0x00007fd3039dc000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007ae410420> (a java.lang.Object) > at > org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371) > - locked <0x00000007ae410420> (a java.lang.Object) > at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740) > at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206) > at > org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352) > at > org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102) > at > org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141) > at > org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204) > at > org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827) > at > org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852) > at > org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840) > at > org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704) > - locked <0x00000007ad1abd88> (a > org.apache.camel.impl.DefaultCamelContext) > at > org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102) > at > com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43) > at > org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > at > org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) > at > org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87) > at > org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) > - locked <0x00000007ae539450> (a java.lang.Object) > > > Example of this situation looks like this: > > package com.example.route; > > import static org.apache.camel.LoggingLevel.ERROR; > import static org.apache.camel.LoggingLevel.WARN; > > import org.apache.camel.Exchange; > import org.apache.camel.Processor; > import org.apache.camel.builder.RouteBuilder; > import org.joda.time.DateTime; > > public class MyAsyncDelayedDeliveryTesting extends RouteBuilder { > > > @Override > public void configure() throws Exception { > > // Set lower timeout for convenience (default 300 seconds). > getContext().getShutdownStrategy().setTimeout(10); > > // Start some route1 cron scheduled. > DateTime now = DateTime.now().plusSeconds(2); > from("quartz2://my_route_quartz_timer?cron=" > +now.getSecondOfMinute()+"+" > +now.getMinuteOfHour()+"+" > +now.getHourOfDay()+"+*+*+?") > .id("scheduling1") > .to("direct://my-route"); > > > // Start other route that shutdowns the context thus shutting > down the > route1. > // Simulate the situation where you want to restart your > route when there > are still > // ongoing redelivery tasks. > now = DateTime.now().plusSeconds(4); > from("quartz2://my_shutdown_route_quartz_timer?cron=" > +now.getSecondOfMinute()+"+" > +now.getMinuteOfHour()+"+" > +now.getHourOfDay()+"+*+*+?") > .id("scheduling2") > .process(new Processor() { > > @Override > public void process(Exchange arg0) throws Exception { > getContext().stop(); > > } > }); > > // Do some logic that causes exception and redeliveries for 1 > minute. > from("direct://my-route") > .id("route1") > .errorHandler(defaultErrorHandler() > .maximumRedeliveries(10) > .redeliveryDelay(1000*6) > // asyncDelayedRedelivery causes the problem. > .asyncDelayedRedelivery() > .retryAttemptedLogLevel(WARN) > .retriesExhaustedLogLevel(ERROR)) > > .process(new Processor() { > > @Override > public void process(Exchange exchange) throws > Exception { > throw new RuntimeException("Error"); > } > }); > } > } > > Full example is visible here: > https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen ----------------- Red Hat, Inc. Email: [email protected] Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
