These is Quartz config in system.properties
Below you can find the stack trace
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreCMT
org.quartz.jobStore.dataSource=MyAppDB
org.quartz.dataSource.MyAppDB.jndiURL=openejb:Resource/MyAppDB
org.quartz.jobStore.nonManagedTXDataSource=MyAppDBNonJta
org.quartz.dataSource.MyAppDBNonJta.jndiURL=openejb:Resource/MyAppDBNonJta
org.quartz.jobStore.isClustered=true
org.quartz.scheduler.instanceId=InstanceA
One note: this is the case when TomEE starts and the DB is down
2012-08-19 20:25:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.1-b02 mixed mode):
"DestroyJavaVM" prio=6 tid=0x0000000008997000 nid=0xeac waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"QuartzScheduler_OpenEJB-TimerService-Scheduler-InstanceA_MisfireHandler" prio=6
tid=0x0000000008995800 nid=0x1064 waiting on condition [0x000000000c76f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3988)
"QuartzScheduler_OpenEJB-TimerService-Scheduler-InstanceA_ClusterManager" prio=8
tid=0x0000000008995000 nid=0xd68 waiting on condition [0x000000000cc0e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3902)
"Timer-0" daemon prio=6 tid=0x0000000008994800 nid=0xf54 in
Object.wait() [0x000000000ca3f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0dcac90> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c0dcac90> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"OpenEJB-TimerService-Scheduler_QuartzSchedulerThread" prio=6
tid=0x0000000008993800 nid=0x12e8 in Object.wait() [0x000000000c87f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0dc9d20> (a java.lang.Object)
at
org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:437)
- locked <0x00000000c0dc9d20> (a java.lang.Object)
"Stateful cache" daemon prio=6 tid=0x0000000008aee800 nid=0x86c waiting
on condition [0x0000000009f9e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c07ed658> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RetryTimer" daemon prio=6 tid=0x000000000868c800 nid=0xd44 in
Object.wait() [0x0000000009ccf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c076c830> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c076c830> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"warmup-4" daemon prio=6 tid=0x0000000007b69800 nid=0xab0 waiting on
condition [0x000000000984e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-3" daemon prio=6 tid=0x0000000007b1a000 nid=0x13cc waiting on
condition [0x000000000972e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-2" daemon prio=6 tid=0x0000000007d86800 nid=0x9c4 waiting on
condition [0x000000000946e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-1" daemon prio=6 tid=0x0000000007d33800 nid=0x360 waiting on
condition [0x000000000930e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"GC Daemon" daemon prio=2 tid=0x0000000007c08800 nid=0x694 in
Object.wait() [0x000000000745f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c024b478> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x00000000c024b478> (a sun.misc.GC$LatencyLock)
"Service Thread" daemon prio=6 tid=0x0000000005ed4800 nid=0xa00 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x0000000005ed0800 nid=0xdf0
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x0000000005ecb000 nid=0xbac
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x0000000005e92800 nid=0xc9c
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000005e91800 nid=0x124c
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x0000000005e08800 nid=0x12b0 in
Object.wait() [0x000000000673f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c068d428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000c068d428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x0000000005dff800 nid=0x12c8 in
Object.wait() [0x000000000640f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c068d160> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000c068d160> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000005df7800 nid=0x494 runnable
"GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000002179000 nid=0x47c
runnable
"GC task thread#1 (ParallelGC)" prio=6 tid=0x000000000217a800 nid=0x1338
runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000005ee7800 nid=0x4c8
waiting on condition
JNI global references: 319
Quartz continues to print this error....
Grave: ClusterManager: Error managing cluster: Failed to obtain DB
connection from data source 'MyAppDBNonJta': java.sql.SQLException:
Could not retrieve datasource via JNDI url
'openejb:Resource/MyAppDBNonJta' javax.naming.NamingException: null
org.quartz.JobPersistenceException: Failed to obtain DB connection from
data source 'MyAppDBNonJta': java.sql.SQLException: Could not retrieve
datasource via JNDI url 'openejb:Resource/MyAppDBNonJta'
javax.naming.NamingException: null [See nested exception:
java.sql.SQLException: Could not retrieve datasource via JNDI url
'openejb:Resource/MyAppDBNonJta' javax.naming.NamingException: null]
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3245)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3870)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3907)
Caused by: java.sql.SQLException: Could not retrieve datasource via JNDI
url 'openejb:Resource/MyAppDBNonJta' javax.naming.NamingException: null
at
org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:163)
at
org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
... 3 more
Il 19/08/2012 19:08, Romain Manni-Bucau ha scritto:
forgot to mention: org.quartz.plugin.shutdownhook.cleanShutdown (see
http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigPlugins )
this last one should be enough
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/19 Romain Manni-Bucau <rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com>>
Hi,
nice to hear you can now go further :)
well some note about your issue:
1) if you can share with us a thread dump (jps to get your java
process id + jstack <id> to get the stack ;)) it can help a lot
2) did you specify the
properties org.quartz.scheduler.makeSchedulerThreadDaemon and
org.quartz.threadPool.makeThreadsDaemons (the last one is relevant
if you specify org.quartz.threadPool.class otherwise by default we
put a thread factory using daemon threads)?
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/18 Enrico Olivelli <eolive...@gmail.com
<mailto:eolive...@gmail.com>>
Hi all,
I'm continuing with tests on Quartz and TomEE and Jobstore on DB.
I found this interesting situation:
- start Tomee with Quartz configured with a JobStore on a
shared DB in order to deploy "clustered" EJB timeouts
- shutdown the database
- try to stop TomEE
The JVM will never shutdown, hung on (I think) Quartz non
deamon threads
Personally I think this is more a Tomcat problem, I never
liked Tomcat shutdown process
I agree that for the container (Tomcat) it is a good thing to
let all thread stop gracefully but is a real world scenario
you would like to be able to stop your Tomcat without the need
of a "kill -KILL"
this scenario cannot be easily reproduced on a JUnit test case
- Enrico
