With this property shutdown now works cleanly
org.quartz.jobStore.makeThreadsDaemons=true
maybe it would be a safe settings to use this property too
org.quartz.scheduler.makeSchedulerThreadDaemon=true
I think that this two properties need to be always set to true for
OpenEJB/TomEE
would you consider to hard-code them in EBJTimerImpl ?
thanks
- Enrico
Il 20/08/2012 10:28, Romain Manni-Bucau ha scritto:
maybe tunning a bit the datasource pool to validate the connection "on
borrow" can help for such cases
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/20 Enrico Olivelli <[email protected]
<mailto:[email protected]>>
I think that Quartz says "Could not retrieve datasource via JNDI
url" instead of saying "could not get a connection from DataSource"
org.quartz.JobPersistenceException: Failed to obtain DB connection
from data source 'MyAppDBNonJta': java.sql.SQLException: Could not
retrieve datasource via JNDI url 'openejb:Resource/MyAppDBNonJta'
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server. [See
nested exception: java.sql.SQLException: Could not retrieve
datasource via JNDI url 'openejb:Resource/MyAppDBNonJta'
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.]
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3801)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.triggersFired(JobStoreSupport.java:2893)
at
org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:326)
Caused by:java.sql.SQLException: Could not retrieve datasource via
JNDI url 'openejb:Resource/MyAppDBNonJta'
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.
at
org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:163)
at
org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
... 3 more
Il 19/08/2012 20:34, Romain Manni-Bucau ha scritto:
first i'd try the previous settings linked to quartz shutdown.
then about the stack it says the datasource can't be found, can
you check your log? or a manual lookup?
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/19 Enrico Olivelli <[email protected]
<mailto:[email protected]>>
These is Quartz config in system.properties
Below you can find the stack trace
org.quartz.jobStore.class =
org.quartz.impl.jdbcjobstore.JobStoreCMT
org.quartz.jobStore.dataSource=MyAppDB
org.quartz.dataSource.MyAppDB.jndiURL=openejb:Resource/MyAppDB
org.quartz.jobStore.nonManagedTXDataSource=MyAppDBNonJta
org.quartz.dataSource.MyAppDBNonJta.jndiURL=openejb:Resource/MyAppDBNonJta
org.quartz.jobStore.isClustered=true
org.quartz.scheduler.instanceId=InstanceA
One note: this is the case when TomEE starts and the DB is down
2012-08-19 20:25:30
Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.1-b02
mixed mode):
"DestroyJavaVM" prio=6 tid=0x0000000008997000 nid=0xeac
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"QuartzScheduler_OpenEJB-TimerService-Scheduler-InstanceA_MisfireHandler"
prio=6 tid=0x0000000008995800 nid=0x1064 waiting on condition
[0x000000000c76f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3988)
"QuartzScheduler_OpenEJB-TimerService-Scheduler-InstanceA_ClusterManager"
prio=8 tid=0x0000000008995000 nid=0xd68 waiting on condition
[0x000000000cc0e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3902)
"Timer-0" daemon prio=6 tid=0x0000000008994800 nid=0xf54 in
Object.wait() [0x000000000ca3f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0dcac90> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000c0dcac90> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"OpenEJB-TimerService-Scheduler_QuartzSchedulerThread" prio=6
tid=0x0000000008993800 nid=0x12e8 in Object.wait()
[0x000000000c87f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0dc9d20> (a java.lang.Object)
at
org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:437)
- locked <0x00000000c0dc9d20> (a java.lang.Object)
"Stateful cache" daemon prio=6 tid=0x0000000008aee800
nid=0x86c waiting on condition [0x0000000009f9e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c07ed658> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RetryTimer" daemon prio=6 tid=0x000000000868c800 nid=0xd44
in Object.wait() [0x0000000009ccf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c076c830> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:503)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c076c830> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"warmup-4" daemon prio=6 tid=0x0000000007b69800 nid=0xab0
waiting on condition [0x000000000984e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-3" daemon prio=6 tid=0x0000000007b1a000 nid=0x13cc
waiting on condition [0x000000000972e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-2" daemon prio=6 tid=0x0000000007d86800 nid=0x9c4
waiting on condition [0x000000000946e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"warmup-1" daemon prio=6 tid=0x0000000007d33800 nid=0x360
waiting on condition [0x000000000930e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0135778> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"GC Daemon" daemon prio=2 tid=0x0000000007c08800 nid=0x694 in
Object.wait() [0x000000000745f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c024b478> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:117)
- locked <0x00000000c024b478> (a sun.misc.GC$LatencyLock)
"Service Thread" daemon prio=6 tid=0x0000000005ed4800
nid=0xa00 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x0000000005ed0800
nid=0xdf0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x0000000005ecb000
nid=0xbac waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x0000000005e92800
nid=0xc9c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000005e91800
nid=0x124c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x0000000005e08800 nid=0x12b0
in Object.wait() [0x000000000673f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c068d428> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000c068d428> (a
java.lang.ref.ReferenceQueue$Lock)
at
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x0000000005dff800
nid=0x12c8 in Object.wait() [0x000000000640f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c068d160> (a
java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000c068d160> (a
java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000005df7800 nid=0x494 runnable
"GC task thread#0 (ParallelGC)" prio=6 tid=0x0000000002179000
nid=0x47c runnable
"GC task thread#1 (ParallelGC)" prio=6 tid=0x000000000217a800
nid=0x1338 runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000005ee7800
nid=0x4c8 waiting on condition
JNI global references: 319
Quartz continues to print this error....
Grave: ClusterManager: Error managing cluster: Failed to
obtain DB connection from data source 'MyAppDBNonJta':
java.sql.SQLException: Could not retrieve datasource via JNDI
url 'openejb:Resource/MyAppDBNonJta'
javax.naming.NamingException: null
org.quartz.JobPersistenceException: Failed to obtain DB
connection from data source 'MyAppDBNonJta':
java.sql.SQLException: Could not retrieve datasource via JNDI
url 'openejb:Resource/MyAppDBNonJta'
javax.naming.NamingException: null [See nested exception:
java.sql.SQLException: Could not retrieve datasource via JNDI
url 'openejb:Resource/MyAppDBNonJta'
javax.naming.NamingException: null]
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3245)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3870)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3907)
Caused by: java.sql.SQLException: Could not retrieve
datasource via JNDI url 'openejb:Resource/MyAppDBNonJta'
javax.naming.NamingException: null
at
org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:163)
at
org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
... 3 more
Il 19/08/2012 19:08, Romain Manni-Bucau ha scritto:
forgot to mention:
org.quartz.plugin.shutdownhook.cleanShutdown (see
http://quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigPlugins
)
this last one should be enough
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/19 Romain Manni-Bucau <[email protected]
<mailto:[email protected]>>
Hi,
nice to hear you can now go further :)
well some note about your issue:
1) if you can share with us a thread dump (jps to get
your java process id + jstack <id> to get the stack ;))
it can help a lot
2) did you specify the
properties org.quartz.scheduler.makeSchedulerThreadDaemon and
org.quartz.threadPool.makeThreadsDaemons (the last one
is relevant if you specify org.quartz.threadPool.class
otherwise by default we put a thread factory using
daemon threads)?
/Romain Manni-Bucau/
/Twitter: @rmannibucau/
/Blog: http://rmannibucau.wordpress.com/
2012/8/18 Enrico Olivelli <[email protected]
<mailto:[email protected]>>
Hi all,
I'm continuing with tests on Quartz and TomEE and
Jobstore on DB.
I found this interesting situation:
- start Tomee with Quartz configured with a JobStore
on a shared DB in order to deploy "clustered" EJB
timeouts
- shutdown the database
- try to stop TomEE
The JVM will never shutdown, hung on (I think)
Quartz non deamon threads
Personally I think this is more a Tomcat problem, I
never liked Tomcat shutdown process
I agree that for the container (Tomcat) it is a good
thing to let all thread stop gracefully but is a
real world scenario you would like to be able to
stop your Tomcat without the need of a "kill -KILL"
this scenario cannot be easily reproduced on a JUnit
test case
- Enrico
