Hi!
I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2)
cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler.
If I want to shutdown/restart routes during DefaultErrorHandlers
redeliveries quartz scheduling gets deadlocked and quietly fails all future
fire times. Without AsyncDelayedDelivery-option this problem does not occur.
I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs
when running the code on jvm (at least on 1.7) without Karaf.
I have configured quite long redelivery times for my routes (e.g. 8 times in
2 hour interval) but at times I would need to be able to restart bundles
running in Karaf even when there would be redeliveries still left (e.g. when
installing new version of the bundle).
This seems like a bug to me but what's your take on this? I've attached few
lines of log and some example route code to the end of this message.
Best regards,
- Jussi
Logs on org.quartz debug level show every few seconds:
[artzScheduler-camel-1_Worker-2] SimpleThreadPool DEBUG
Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down
and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
"DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007ae410420> (a java.lang.Object)
at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
- locked <0x00000007ae410420> (a java.lang.Object)
at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
at
org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at
org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
at
org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
at
org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
- locked <0x00000007ad1abd88> (a
org.apache.camel.impl.DefaultCamelContext)
at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
at
com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000007ae539450> (a java.lang.Object)
Example of this situation looks like this:
package com.example.route;
import static org.apache.camel.LoggingLevel.ERROR;
import static org.apache.camel.LoggingLevel.WARN;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.joda.time.DateTime;
public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
@Override
public void configure() throws Exception {
// Set lower timeout for convenience (default 300 seconds).
getContext().getShutdownStrategy().setTimeout(10);
// Start some route1 cron scheduled.
DateTime now = DateTime.now().plusSeconds(2);
from("quartz2://my_route_quartz_timer?cron="
+now.getSecondOfMinute()+"+"
+now.getMinuteOfHour()+"+"
+now.getHourOfDay()+"+*+*+?")
.id("scheduling1")
.to("direct://my-route");
// Start other route that shutdowns the context thus shutting
down the
route1.
// Simulate the situation where you want to restart your route
when there
are still
// ongoing redelivery tasks.
now = DateTime.now().plusSeconds(4);
from("quartz2://my_shutdown_route_quartz_timer?cron="
+now.getSecondOfMinute()+"+"
+now.getMinuteOfHour()+"+"
+now.getHourOfDay()+"+*+*+?")
.id("scheduling2")
.process(new Processor() {
@Override
public void process(Exchange arg0) throws Exception {
getContext().stop();
}
});
// Do some logic that causes exception and redeliveries for 1
minute.
from("direct://my-route")
.id("route1")
.errorHandler(defaultErrorHandler()
.maximumRedeliveries(10)
.redeliveryDelay(1000*6)
// asyncDelayedRedelivery causes the problem.
.asyncDelayedRedelivery()
.retryAttemptedLogLevel(WARN)
.retriesExhaustedLogLevel(ERROR))
.process(new Processor() {
@Override
public void process(Exchange exchange) throws Exception
{
throw new RuntimeException("Error");
}
});
}
}
Full example is visible here:
https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem
--
View this message in context:
http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
Sent from the Camel - Users mailing list archive at Nabble.com.
