Memory leak - when an exception is thrown during the scheduling phase
Hello, I have observed a memory leak for Quartz worker thread in case an exception is thrown during the scheduling phase when schedule is set to past time. I have observed this issue with camel-quart 2.10.4, 2.11.0, 2.11.2 versions. For ex, assuming today is 4-Oct; If a schedule is provided for yesterday (i.e 3-Oct) then below sequence happen: 1. Quartz create worker thread for new schedule [generally TEN Threads with name 'DefaultQuartzScheduler'] 2. Camel-quartz increment jobCounter for scheduler context [using QuartzComponent.incrementJobCounter()] 3. Camel-quartz tries to create new schedule [using Scheduler.scheduleJob()] 4. scheduleJob() throws SchedulerException with the message - 'Based on configured schedule, the given trigger will never fire' 5. Camel-quartz does NOT decrement the job counter and jobCounter value remains '1' 6. Later Camel-Quartz stops the route and call QuartzComponent.doStop() 7. In doStop(), Scheduler.shutdown() supposed to be called so that those TEN threads are removed but to call shutdown() the pre-condition is that the jobCounter should be '0'. In this case jobCounter is '1' and Scheduler.shutdown() will NOT be called and those TEN threads are NOT removed. This is memory leak. Please find the stack trace [with Camel-Quartz 2.11.2 and org.apache.servicemix.bundles.quartz-1.8.6_1 which wrapped Quartz Scheduler as OSGi bundle] when SchedulerException is thrown at step 4 in above sequence. Let me know if you want me to create JIRA issue. Thanks, Nurali ** Stack Trace: 15:55:22.432 ERROR [Blueprint Extender: 1] o.a.c.b.BlueprintCamelContext [BlueprintCamelContext.java:139] - Error occurred during starting Camel: CamelContext(sfsftest35) due Based on configured schedule, the given trigger will never fire. org.quartz.SchedulerException: Based on configured schedule, the given trigger will never fire. at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:793) ~[na:na] at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) ~[na:na] at org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) ~[na:na] at org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) ~[na:na] at org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) ~[na:na] at org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) ~[na:na] at org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) ~[na:na] at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) ~[na:na] at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) ~[na:na] at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1468) ~[na:na] at org.apache.camel.blueprint.BlueprintCamelContext.start(BlueprintCamelContext.java:167) [bundlefile:2.11.2] at org.apache.camel.blueprint.BlueprintCamelContext.maybeStart(BlueprintCamelContext.java:199) [bundlefile:2.11.2] at org.apache.camel.blueprint.BlueprintCamelContext.serviceChanged(BlueprintCamelContext.java:137) [bundlefile:2.11.2] at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.jav
Re: Memory leak - when an exception is thrown during the scheduling phase
Create a JIRA. Looks like you may have a patch in mind? Do you have test cases? On Friday, October 4, 2013, Nurali Techie wrote: > Hello, > > > > I have observed a memory leak for Quartz worker thread in case an > exception is thrown during the scheduling phase when schedule is set to > past time. > I have observed this issue with camel-quart 2.10.4, 2.11.0, 2.11.2 > versions. > > For ex, assuming today is 4-Oct; If a schedule is provided for yesterday > (i.e 3-Oct) then below sequence happen: > > 1. Quartz create worker thread for new schedule [generally TEN Threads > with name 'DefaultQuartzScheduler'] > 2. Camel-quartz increment jobCounter for scheduler context [using > QuartzComponent.incrementJobCounter()] > 3. Camel-quartz tries to create new schedule > [using Scheduler.scheduleJob()] > 4. scheduleJob() throws SchedulerException with the message - 'Based on > configured schedule, the given trigger will never fire' > 5. Camel-quartz does NOT decrement the job counter and jobCounter value > remains '1' > 6. Later Camel-Quartz stops the route and call QuartzComponent.doStop() > 7. In doStop(), Scheduler.shutdown() supposed to be called so that those > TEN threads are removed but to call shutdown() the pre-condition is that > the jobCounter should be '0'. In this case jobCounter is '1' > and Scheduler.shutdown() will NOT be called and those TEN threads are NOT > removed. This is memory leak. > > Please find the stack trace [with Camel-Quartz 2.11.2 and > org.apache.servicemix.bundles.quartz-1.8.6_1 which wrapped Quartz Scheduler > as OSGi bundle] when SchedulerException is thrown at step 4 in above > sequence. > > Let me know if you want me to create JIRA issue. > > Thanks, > Nurali > > ** Stack Trace: > > 15:55:22.432 ERROR [Blueprint Extender: 1] o.a.c.b.BlueprintCamelContext > [BlueprintCamelContext.java:139] - Error occurred during starting Camel: > CamelContext(sfsftest35) due Based on configured schedule, the given > trigger will never fire. > org.quartz.SchedulerException: Based on configured schedule, the given > trigger will never fire. > at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:793) > ~[na:na] > at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) ~[na:na] > at > org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) > ~[na:na] > at > org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) > ~[na:na] > at > org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) > ~[na:na] > at > org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) > ~[na:na] > at > org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) > ~[na:na] > at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) > ~[na:na] > at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) > ~[na:na] > at > org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1468) > ~[na:na] > at > org.apache.camel.blueprint.BlueprintCamelContext.start(BlueprintCamelContext.java:167) > [bundlefile:2.11.2] > at > org.apache.camel.blueprint.BlueprintCamelContext.maybeStart(BlueprintCamelContext.java:199) > [bundlefile:2.11.2] > at > org.apache.camel.blueprint.BlueprintCamelContext.serviceChanged(BlueprintCamelContext.java:137) > [bundlefile:2.11.2] > at > org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107) > [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] > at > org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861) > [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] > at > org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) > [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] > at > org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) > [org.eclipse.org.eclipse.osgi-3.8.0.v20120529-1548.jar:na] > at > org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) > [org.eclipse.org.eclipse.osgi-3.8.0
Re: Memory leak - when an exception is thrown during the scheduling phase
Thanks James for the response .. I have created Jira issue. https://issues.apache.org/jira/browse/CAMEL-6832 Yes, I know which code causing this issue and how to fix that. But I don't know other usecases (around the code where the fix needed). I will spend some time and will update my finding/fix in Jira issue. @All, Please follow Jira issue for further details. Thanks, Nurali From: James Carman To: "users@camel.apache.org" ; Nurali Techie Sent: Friday, October 4, 2013 4:32 PM Subject: Re: Memory leak - when an exception is thrown during the scheduling phase Create a JIRA. Looks like you may have a patch in mind? Do you have test cases? On Friday, October 4, 2013, Nurali Techie wrote: Hello, > > > >I have observed a memory leak for Quartz worker thread in case an exception is >thrown during the scheduling phase when schedule is set to past time. >I have observed this issue with camel-quart 2.10.4, 2.11.0, 2.11.2 versions. > >For ex, assuming today is 4-Oct; If a schedule is provided for yesterday (i.e >3-Oct) then below sequence happen: > >1. Quartz create worker thread for new schedule [generally TEN Threads with >name 'DefaultQuartzScheduler'] >2. Camel-quartz increment jobCounter for scheduler context [using >QuartzComponent.incrementJobCounter()] >3. Camel-quartz tries to create new schedule [using Scheduler.scheduleJob()] >4. scheduleJob() throws SchedulerException with the message - 'Based on >configured schedule, the given trigger will never fire' >5. Camel-quartz does NOT decrement the job counter and jobCounter value >remains '1' >6. Later Camel-Quartz stops the route and call QuartzComponent.doStop() >7. In doStop(), Scheduler.shutdown() supposed to be called so that those TEN >threads are removed but to call shutdown() the pre-condition is that the >jobCounter should be '0'. In this case jobCounter is '1' and >Scheduler.shutdown() will NOT be called and those TEN threads are NOT removed. > This is memory leak. > >Please find the stack trace [with Camel-Quartz 2.11.2 and >org.apache.servicemix.bundles.quartz-1.8.6_1 which wrapped Quartz Scheduler as >OSGi bundle] when SchedulerException is thrown at step 4 in above sequence. > >Let me know if you want me to create JIRA issue. > >Thanks, >Nurali > >** Stack Trace: > >15:55:22.432 ERROR [Blueprint Extender: 1] o.a.c.b.BlueprintCamelContext >[BlueprintCamelContext.java:139] - Error occurred during starting Camel: >CamelContext(sfsftest35) due Based on configured schedule, the given trigger >will never fire. >org.quartz.SchedulerException: Based on configured schedule, the given trigger >will never fire. >at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:793) >~[na:na] >at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) ~[na:na] >at >org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) > ~[na:na] >at >org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) > ~[na:na] >at >org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) > ~[na:na] >at >org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) > ~[na:na] >at >org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) > ~[na:na] >at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) >~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) > ~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) > ~[na:na] >at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) >~[na:na] >at >org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:1468) >~[na:na] >at >org.apache.camel.blueprint.BlueprintCamelContext.start(BlueprintCamelContext.java:167) > [bundlefile:2.11.2] >at >org.apache.camel.blueprint.BlueprintCamelContext.maybeStart(BlueprintCamelCo
