right, https://issues.apache.org/jira/browse/TOMEE-389
*Romain Manni-Bucau* *Twitter: @rmannibucau* *Blog: http://rmannibucau.wordpress.com* 2012/8/20 Enrico Olivelli <[email protected]> > With this property shutdown now works cleanly > org.quartz.jobStore.**makeThreadsDaemons=true > maybe it would be a safe settings to use this property too > org.quartz.scheduler.**makeSchedulerThreadDaemon=true > > I think that this two properties need to be always set to true for > OpenEJB/TomEE > would you consider to hard-code them in EBJTimerImpl ? > > thanks > - Enrico > > > Il 20/08/2012 10:28, Romain Manni-Bucau ha scritto: > >> maybe tunning a bit the datasource pool to validate the connection "on >> borrow" can help for such cases >> >> /Romain Manni-Bucau/ >> /Twitter: @rmannibucau/ >> /Blog: http://rmannibucau.wordpress.**com/<http://rmannibucau.wordpress.com/> >> >> >> >> >> 2012/8/20 Enrico Olivelli <[email protected] <mailto: >> [email protected]>> >> >> >> I think that Quartz says "Could not retrieve datasource via JNDI >> url" instead of saying "could not get a connection from DataSource" >> >> >> >> org.quartz.**JobPersistenceException: Failed to obtain DB connection >> from data source 'MyAppDBNonJta': java.sql.SQLException: Could not >> retrieve datasource via JNDI url 'openejb:Resource/**MyAppDBNonJta' >> com.mysql.jdbc.exceptions.**jdbc4.CommunicationsException: >> Communications link failure >> >> The last packet sent successfully to the server was 0 milliseconds >> ago. The driver has not received any packets from the server. [See >> nested exception: java.sql.SQLException: Could not retrieve >> datasource via JNDI url 'openejb:Resource/**MyAppDBNonJta' >> com.mysql.jdbc.exceptions.**jdbc4.CommunicationsException: >> Communications link failure >> >> The last packet sent successfully to the server was 0 milliseconds >> ago. The driver has not received any packets from the server.] >> at >> org.quartz.impl.jdbcjobstore.**JobStoreCMT.** >> getNonManagedTXConnection(**JobStoreCMT.java:168) >> at >> org.quartz.impl.jdbcjobstore.**JobStoreSupport.** >> executeInNonManagedTXLock(**JobStoreSupport.java:3801) >> at >> org.quartz.impl.jdbcjobstore.**JobStoreSupport.triggersFired(** >> JobStoreSupport.java:2893) >> at >> org.quartz.core.**QuartzSchedulerThread.run(** >> QuartzSchedulerThread.java:**326) >> Caused by:java.sql.SQLException: Could not retrieve datasource via >> JNDI url 'openejb:Resource/**MyAppDBNonJta' >> com.mysql.jdbc.exceptions.**jdbc4.CommunicationsException: >> Communications link failure >> >> The last packet sent successfully to the server was 0 milliseconds >> ago. The driver has not received any packets from the server. >> >> at >> org.quartz.utils.**JNDIConnectionProvider.**getConnection(** >> JNDIConnectionProvider.java:**163) >> at >> org.quartz.utils.**DBConnectionManager.**getConnection(** >> DBConnectionManager.java:108) >> at >> org.quartz.impl.jdbcjobstore.**JobStoreCMT.** >> getNonManagedTXConnection(**JobStoreCMT.java:165) >> ... 3 more >> >> >> Il 19/08/2012 20:34, Romain Manni-Bucau ha scritto: >> >>> first i'd try the previous settings linked to quartz shutdown. >>> >>> then about the stack it says the datasource can't be found, can >>> you check your log? or a manual lookup? >>> >>> /Romain Manni-Bucau/ >>> /Twitter: @rmannibucau/ >>> /Blog: >>> http://rmannibucau.wordpress.**com/<http://rmannibucau.wordpress.com/> >>> >>> >>> >>> >>> 2012/8/19 Enrico Olivelli <[email protected] >>> <mailto:[email protected]>> >>> >>> >>> These is Quartz config in system.properties >>> Below you can find the stack trace >>> >>> org.quartz.jobStore.class = >>> org.quartz.impl.jdbcjobstore.**JobStoreCMT >>> org.quartz.jobStore.**dataSource=MyAppDB >>> org.quartz.dataSource.MyAppDB.**jndiURL=openejb:Resource/** >>> MyAppDB >>> org.quartz.jobStore.**nonManagedTXDataSource=**MyAppDBNonJta >>> org.quartz.dataSource.**MyAppDBNonJta.jndiURL=openejb:** >>> Resource/MyAppDBNonJta >>> org.quartz.jobStore.**isClustered=true >>> org.quartz.scheduler.**instanceId=InstanceA >>> >>> One note: this is the case when TomEE starts and the DB is down >>> >>> >>> 2012-08-19 20:25:30 >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.1-b02 >>> mixed mode): >>> >>> "DestroyJavaVM" prio=6 tid=0x0000000008997000 nid=0xeac >>> waiting on condition [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "QuartzScheduler_OpenEJB-**TimerService-Scheduler-** >>> InstanceA_MisfireHandler" >>> prio=6 tid=0x0000000008995800 nid=0x1064 waiting on condition >>> [0x000000000c76f000] >>> java.lang.Thread.State: TIMED_WAITING (sleeping) >>> at java.lang.Thread.sleep(Native Method) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreSupport$** >>> MisfireHandler.run(**JobStoreSupport.java:3988) >>> >>> "QuartzScheduler_OpenEJB-**TimerService-Scheduler-** >>> InstanceA_ClusterManager" >>> prio=8 tid=0x0000000008995000 nid=0xd68 waiting on condition >>> [0x000000000cc0e000] >>> java.lang.Thread.State: TIMED_WAITING (sleeping) >>> at java.lang.Thread.sleep(Native Method) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreSupport$** >>> ClusterManager.run(**JobStoreSupport.java:3902) >>> >>> "Timer-0" daemon prio=6 tid=0x0000000008994800 nid=0xf54 in >>> Object.wait() [0x000000000ca3f000] >>> java.lang.Thread.State: TIMED_WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c0dcac90> (a java.util.TaskQueue) >>> at java.util.TimerThread.**mainLoop(Timer.java:552) >>> - locked <0x00000000c0dcac90> (a java.util.TaskQueue) >>> at java.util.TimerThread.run(**Timer.java:505) >>> >>> "OpenEJB-TimerService-**Scheduler_**QuartzSchedulerThread" >>> prio=6 >>> tid=0x0000000008993800 nid=0x12e8 in Object.wait() >>> [0x000000000c87f000] >>> java.lang.Thread.State: TIMED_WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c0dc9d20> (a java.lang.Object) >>> at >>> org.quartz.core.**QuartzSchedulerThread.run(** >>> QuartzSchedulerThread.java:**437) >>> - locked <0x00000000c0dc9d20> (a java.lang.Object) >>> >>> "Stateful cache" daemon prio=6 tid=0x0000000008aee800 >>> nid=0x86c waiting on condition [0x0000000009f9e000] >>> java.lang.Thread.State: TIMED_WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000000c07ed658> (a >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject) >>> at >>> java.util.concurrent.locks.**LockSupport.parkNanos(** >>> LockSupport.java:226) >>> at >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject.awaitNanos(**AbstractQueuedSynchronizer.**java:2082) >>> at >>> java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> DelayedWorkQueue.take(**ScheduledThreadPoolExecutor.**java:1090) >>> at >>> java.util.concurrent.**ScheduledThreadPoolExecutor$** >>> DelayedWorkQueue.take(**ScheduledThreadPoolExecutor.**java:807) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.getTask(** >>> ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.runWorker(** >>> ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.**java:722) >>> >>> "RetryTimer" daemon prio=6 tid=0x000000000868c800 nid=0xd44 >>> in Object.wait() [0x0000000009ccf000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c076c830> (a java.util.TaskQueue) >>> at java.lang.Object.wait(Object.**java:503) >>> at java.util.TimerThread.**mainLoop(Timer.java:526) >>> - locked <0x00000000c076c830> (a java.util.TaskQueue) >>> at java.util.TimerThread.run(**Timer.java:505) >>> >>> "warmup-4" daemon prio=6 tid=0x0000000007b69800 nid=0xab0 >>> waiting on condition [0x000000000984e000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000000c0135778> (a >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject) >>> at >>> java.util.concurrent.locks.**LockSupport.park(LockSupport.** >>> java:186) >>> at >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject.await(**AbstractQueuedSynchronizer.**java:2043) >>> at >>> java.util.concurrent.**LinkedBlockingQueue.take(** >>> LinkedBlockingQueue.java:442) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.getTask(** >>> ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.runWorker(** >>> ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.**java:722) >>> >>> "warmup-3" daemon prio=6 tid=0x0000000007b1a000 nid=0x13cc >>> waiting on condition [0x000000000972e000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000000c0135778> (a >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject) >>> at >>> java.util.concurrent.locks.**LockSupport.park(LockSupport.** >>> java:186) >>> at >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject.await(**AbstractQueuedSynchronizer.**java:2043) >>> at >>> java.util.concurrent.**LinkedBlockingQueue.take(** >>> LinkedBlockingQueue.java:442) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.getTask(** >>> ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.runWorker(** >>> ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.**java:722) >>> >>> "warmup-2" daemon prio=6 tid=0x0000000007d86800 nid=0x9c4 >>> waiting on condition [0x000000000946e000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000000c0135778> (a >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject) >>> at >>> java.util.concurrent.locks.**LockSupport.park(LockSupport.** >>> java:186) >>> at >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject.await(**AbstractQueuedSynchronizer.**java:2043) >>> at >>> java.util.concurrent.**LinkedBlockingQueue.take(** >>> LinkedBlockingQueue.java:442) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.getTask(** >>> ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.runWorker(** >>> ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.**java:722) >>> >>> "warmup-1" daemon prio=6 tid=0x0000000007d33800 nid=0x360 >>> waiting on condition [0x000000000930e000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for <0x00000000c0135778> (a >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject) >>> at >>> java.util.concurrent.locks.**LockSupport.park(LockSupport.** >>> java:186) >>> at >>> java.util.concurrent.locks.**AbstractQueuedSynchronizer$** >>> ConditionObject.await(**AbstractQueuedSynchronizer.**java:2043) >>> at >>> java.util.concurrent.**LinkedBlockingQueue.take(** >>> LinkedBlockingQueue.java:442) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.getTask(** >>> ThreadPoolExecutor.java:1043) >>> at >>> java.util.concurrent.**ThreadPoolExecutor.runWorker(** >>> ThreadPoolExecutor.java:1103) >>> at >>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(** >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.**java:722) >>> >>> "GC Daemon" daemon prio=2 tid=0x0000000007c08800 nid=0x694 in >>> Object.wait() [0x000000000745f000] >>> java.lang.Thread.State: TIMED_WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c024b478> (a sun.misc.GC$LatencyLock) >>> at sun.misc.GC$Daemon.run(GC.**java:117) >>> - locked <0x00000000c024b478> (a sun.misc.GC$LatencyLock) >>> >>> "Service Thread" daemon prio=6 tid=0x0000000005ed4800 >>> nid=0xa00 runnable [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "C2 CompilerThread1" daemon prio=10 tid=0x0000000005ed0800 >>> nid=0xdf0 waiting on condition [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "C2 CompilerThread0" daemon prio=10 tid=0x0000000005ecb000 >>> nid=0xbac waiting on condition [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Attach Listener" daemon prio=10 tid=0x0000000005e92800 >>> nid=0xc9c waiting on condition [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Signal Dispatcher" daemon prio=10 tid=0x0000000005e91800 >>> nid=0x124c runnable [0x0000000000000000] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Finalizer" daemon prio=8 tid=0x0000000005e08800 nid=0x12b0 >>> in Object.wait() [0x000000000673f000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c068d428> (a >>> java.lang.ref.ReferenceQueue$**Lock) >>> at >>> java.lang.ref.ReferenceQueue.**remove(ReferenceQueue.java:**135) >>> - locked <0x00000000c068d428> (a >>> java.lang.ref.ReferenceQueue$**Lock) >>> at >>> java.lang.ref.ReferenceQueue.**remove(ReferenceQueue.java:**151) >>> at >>> java.lang.ref.Finalizer$**FinalizerThread.run(Finalizer.** >>> java:177) >>> >>> "Reference Handler" daemon prio=10 tid=0x0000000005dff800 >>> nid=0x12c8 in Object.wait() [0x000000000640f000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on <0x00000000c068d160> (a >>> java.lang.ref.Reference$Lock) >>> at java.lang.Object.wait(Object.**java:503) >>> at >>> java.lang.ref.Reference$**ReferenceHandler.run(** >>> Reference.java:133) >>> - locked <0x00000000c068d160> (a >>> java.lang.ref.Reference$Lock) >>> >>> "VM Thread" prio=10 tid=0x0000000005df7800 nid=0x494 runnable >>> >>> "GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000002179000 >>> nid=0x47c runnable >>> >>> "GC task thread#1 (ParallelGC)" prio=6 tid=0x000000000217a800 >>> nid=0x1338 runnable >>> >>> "VM Periodic Task Thread" prio=10 tid=0x0000000005ee7800 >>> nid=0x4c8 waiting on condition >>> >>> JNI global references: 319 >>> >>> >>> >>> >>> Quartz continues to print this error.... >>> >>> >>> Grave: ClusterManager: Error managing cluster: Failed to >>> obtain DB connection from data source 'MyAppDBNonJta': >>> java.sql.SQLException: Could not retrieve datasource via JNDI >>> url 'openejb:Resource/**MyAppDBNonJta' >>> javax.naming.NamingException: null >>> org.quartz.**JobPersistenceException: Failed to obtain DB >>> connection from data source 'MyAppDBNonJta': >>> java.sql.SQLException: Could not retrieve datasource via JNDI >>> url 'openejb:Resource/**MyAppDBNonJta' >>> javax.naming.NamingException: null [See nested exception: >>> java.sql.SQLException: Could not retrieve datasource via JNDI >>> url 'openejb:Resource/**MyAppDBNonJta' >>> javax.naming.NamingException: null] >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreCMT.** >>> getNonManagedTXConnection(**JobStoreCMT.java:168) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreSupport.doCheckin(** >>> JobStoreSupport.java:3245) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreSupport$** >>> ClusterManager.manage(**JobStoreSupport.java:3870) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreSupport$** >>> ClusterManager.run(**JobStoreSupport.java:3907) >>> Caused by: java.sql.SQLException: Could not retrieve >>> datasource via JNDI url 'openejb:Resource/**MyAppDBNonJta' >>> javax.naming.NamingException: null >>> at >>> org.quartz.utils.**JNDIConnectionProvider.**getConnection(** >>> JNDIConnectionProvider.java:**163) >>> at >>> org.quartz.utils.**DBConnectionManager.**getConnection(** >>> DBConnectionManager.java:108) >>> at >>> org.quartz.impl.jdbcjobstore.**JobStoreCMT.** >>> getNonManagedTXConnection(**JobStoreCMT.java:165) >>> ... 3 more >>> >>> >>> >>> >>> >>> >>> >>> >>> Il 19/08/2012 19:08, Romain Manni-Bucau ha scritto: >>> >>>> forgot to mention: >>>> org.quartz.plugin.**shutdownhook.cleanShutdown (see >>>> http://quartz-scheduler.org/**documentation/quartz-2.x/** >>>> configuration/ConfigPlugins<http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigPlugins>) >>>> >>>> >>>> this last one should be enough >>>> >>>> /Romain Manni-Bucau/ >>>> /Twitter: @rmannibucau/ >>>> /Blog: >>>> http://rmannibucau.wordpress.**com/<http://rmannibucau.wordpress.com/> >>>> >>>> >>>> >>>> >>>> 2012/8/19 Romain Manni-Bucau <[email protected] >>>> <mailto:[email protected]>**> >>>> >>>> >>>> Hi, >>>> >>>> nice to hear you can now go further :) >>>> >>>> well some note about your issue: >>>> 1) if you can share with us a thread dump (jps to get >>>> your java process id + jstack <id> to get the stack ;)) >>>> it can help a lot >>>> 2) did you specify the >>>> properties org.quartz.scheduler.**makeSchedulerThreadDaemon >>>> and >>>> org.quartz.threadPool.**makeThreadsDaemons (the last one >>>> is relevant if you specify org.quartz.threadPool.class >>>> otherwise by default we put a thread factory using >>>> daemon threads)? >>>> >>>> /Romain Manni-Bucau/ >>>> /Twitter: @rmannibucau/ >>>> /Blog: >>>> http://rmannibucau.wordpress.**com/<http://rmannibucau.wordpress.com/> >>>> >>>> >>>> >>>> >>>> 2012/8/18 Enrico Olivelli <[email protected] >>>> <mailto:[email protected]>> >>>> >>>> >>>> Hi all, >>>> I'm continuing with tests on Quartz and TomEE and >>>> Jobstore on DB. >>>> I found this interesting situation: >>>> - start Tomee with Quartz configured with a JobStore >>>> on a shared DB in order to deploy "clustered" EJB >>>> timeouts >>>> - shutdown the database >>>> - try to stop TomEE >>>> >>>> The JVM will never shutdown, hung on (I think) >>>> Quartz non deamon threads >>>> >>>> Personally I think this is more a Tomcat problem, I >>>> never liked Tomcat shutdown process >>>> I agree that for the container (Tomcat) it is a good >>>> thing to let all thread stop gracefully but is a >>>> real world scenario you would like to be able to >>>> stop your Tomcat without the need of a "kill -KILL" >>>> >>>> this scenario cannot be easily reproduced on a JUnit >>>> test case >>>> >>>> - Enrico >>>> >>>> >>>> >>>> >>>> >>> >>> >> >> > >
