Hi all, I’m running a stand-alone camel app with an embedded activemq broker and am running into unclean shutdowns. When shutting down, I get a graceful shutdown of the camel context, and then an InterruptedExcpetion from KahaDB when ActiveMQ attempts shutdown afterwards.
[ Thread-6] MainSupport INFO Apache Camel 2.14.2 stopping [ Thread-6] SpringCamelContext INFO Apache Camel 2.14.2 (CamelContext: camel) is shutting down [ Thread-6] DefaultShutdownStrategy INFO Starting to graceful shutdown 2 routes (timeout 300 seconds) [amel) thread #2 - ShutdownTask] DefaultShutdownStrategy INFO Route: route2 shutdown complete, was consuming from: Endpoint[file:///home/vagrant/fedora3data/objectStore?noop=true&recursive=true&sendEmptyMessageWhenIdle=true] [amel) thread #2 - ShutdownTask] DefaultShutdownStrategy INFO Route: route1 shutdown complete, was consuming from: Endpoint[activemq://foxml] [ Thread-6] DefaultShutdownStrategy INFO Graceful shutdown of 2 routes completed in 0 seconds [ Thread-6] SpringCamelContext INFO Apache Camel 2.14.2 (CamelContext: camel) uptime 3.128 seconds [ Thread-6] SpringCamelContext INFO Apache Camel 2.14.2 (CamelContext: camel) is shutdown in 0.494 seconds [ Thread-6] BrokerService INFO Apache ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) is shutting down [ Thread-6] TransportConnector INFO Connector vm stopped [ Thread-6] PListStoreImpl INFO PListStore:[/home/vagrant/spring/migration-utils/src/data/myBroker/tmp_storage] stopped [ Thread-6] KahaDBStore INFO Stopping async queue tasks [ Thread-6] KahaDBStore ERROR Could not stop service: KahaDB:[/home/vagrant/spring/migration-utils/src/data/myBroker/KahaDB]. Reason: java.lang.InterruptedException java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:588) at org.apache.activemq.store.kahadb.KahaDBStore.doStop(KahaDBStore.java:247) at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71) at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStop(KahaDBPersistenceAdapter.java:250) at org.apache.activemq.util.ServiceSupport.stop(ServiceSupport.java:71) at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:792) at org.apache.activemq.xbean.XBeanBrokerService.stop(XBeanBrokerService.java:122) at org.apache.activemq.xbean.XBeanBrokerService.destroy(XBeanBrokerService.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:350) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:538) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:514) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:831) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:483) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1092) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1066) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1012) at org.apache.camel.util.IOHelper.close(IOHelper.java:326) at org.apache.camel.util.IOHelper.close(IOHelper.java:390) at org.apache.camel.spring.Main.doStop(Main.java:183) at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102) at org.fcrepo.migration.processors.ShutdownProcessor$1.run(ShutdownProcessor.java:23) The shutdown does eventually continue and complete after hanging for a bit, with the following message: [ Thread-6] BrokerService INFO Apache ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) uptime 4.065 seconds [ Thread-6] BrokerService INFO Apache ActiveMQ 5.11.1 (myBroker, ID:islandora-51066-1427724041540-0:1) is shutdown [ Thread-6] DisposableBeanAdapter WARN Invocation of destroy method 'destroy' failed on bean with name 'broker': java.lang.InterruptedException [WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] was interrupted but is still alive after waiting at least 14995msecs [WARNING] thread Thread[KahaDB Scheduler,5,org.apache.camel.spring.Main] will linger despite being asked to die via interruption [WARNING] thread Thread[ConcurrentQueueStoreAndDispatch,5,org.apache.camel.spring.Main] will linger despite being asked to die via interruption [WARNING] NOTE: 2 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied. [WARNING] Couldn't destroy threadgroup org.apache.camel.maven.RunMojo$IsolatedThreadGroup[name=org.apache.camel.spring.Main,maxpri=10] java.lang.IllegalThreadStateException The interrupted exception makes sense. I’m just wondering what’s the best approach to handling this properly? I’m shutting down the application using another thread as per http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html. Is the problem perhaps how I’m shutting down the Spring context by calling stop on Main? Is there a better way to stop a command-line camel app to ensure things get handled in order and wait for the broker and persistence services to shutdown cleanly? I’ve personally tried shutting down the camel context manually before the spring context, as well as shutdown hooks on the broker bean. Nothing seems to be working. Here’s my shutdown code: package org.fcrepo.migration.processors; import org.apache.camel.Exchange; import org.apache.camel.Processor; import org.apache.camel.spring.Main; /** * Shuts down the application. * * @author Daniel Lamb */ public class ShutdownProcessor implements Processor { private Thread stop; @Override public void process(final Exchange exchange) throws Exception { if (stop == null) { stop = new Thread() { @Override public void run() { try { Main.getInstance().stop(); } catch (Exception e) { // ignore } } }; } stop.start(); } } Here’s my configs (pretty much ripped straight out of the camel activemq-tomcat example: Camel: <beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <!-- config --> <bean id="bridgePropertyPlaceholder" class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer"> <property name="location" value="classpath:migration.properties"/> </bean> <!-- activemq --> <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="brokerURL" value="vm://myBroker?create=false&waitForStart=5000"/> </bean> <camelContext xmlns="http://camel.apache.org/schema/spring" id="camel" shutdownRunningTask="CompleteAllTasks"> <package>org.fcrepo.migration.routes</package> </camelContext> </beans> Activemq: <beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <broker xmlns="http://activemq.apache.org/schema/core" id="broker" brokerName="myBroker" useShutdownHook=“false" persistent="true" dataDirectory="${activemqData}"> <transportConnectors> <transportConnector name="vm" uri="vm://myBroker"/> </transportConnectors> </broker> </beans> Any advice/experience would be greatly appreciated. ~Danny