Hi,
Can you please provide some pointers on debugging this issue:
We have a Spring application that has a jackrabbit repository.
We also use Quartz within this Spring application for some crons.
I see around 63 to 73 connections to the remote mysql server when the app is
running in production
All of them are coming from the tomcat process running our application.
How can I find which parts of the application is making all these connections?
Our application had run out of memory .
The exception is pasted below. It seems to be pointing at Mysql communication
failure.
Please also find attached the 87 threads under tomcat that were logged near the
error.
tomcat settings:
max_heap_size_mb 2048
min_heap_size_mb 512
Logs:
2013-06-05 12:12:27,616 [ClusterNode-node1] ERROR
org.apache.jackrabbit.core.cluster.ClusterNode - Periodic sync of journal
failed: Couldn't sync the cluster node
org.apache.jackrabbit.core.cluster.ClusterException: Couldn't sync the cluster
node
at org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:331)
at org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:293)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
The last packet successfully received from the server was 362,492 milliseconds
ago. The last packet sent successfully to the server was 19,111 milliseconds
ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3102)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618)
at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5000)
at
org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
at
org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:328)
at
org.apache.jackrabbit.core.util.db.ConnectionHelper.startBatch(ConnectionHelper.java:240)
at
org.apache.jackrabbit.core.journal.DatabaseJournal.startBatch(DatabaseJournal.java:506)
at
org.apache.jackrabbit.core.journal.DatabaseJournal.doSync(DatabaseJournal.java:450)
at
org.apache.jackrabbit.core.journal.AbstractJournal.internalSync(AbstractJournal.java:206)
at
org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:189)
at org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:328)
... 2 more
Caused by: java.io.EOFException: Can not read response from server. Expected to
read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2552)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
... 17 more
2013-06-05 12:12:29,934
[QuartzScheduler_org.springframework.scheduling.quartz.SchedulerFactoryBean#0-bf1-rmaicic1.ads.bf1.yahoo.com1370389426455_MisfireHandler]
ERROR org.springframework.scheduling.quartz.LocalDataSourceJobStore -
MisfireHandler: Error handling misfires: Failed to obtain DB connection from
data source
'springNonTxDataSource.org.springframework.scheduling.quartz.SchedulerFactoryBean#0':
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
org.quartz.JobPersistenceException: Failed to obtain DB connection from data
source
'springNonTxDataSource.org.springframework.scheduling.quartz.SchedulerFactoryBean#0':
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space [See nested
exception: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space]
at
org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3132)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3923)
at
org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3943)
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
Thanks
-Priyanka
acl%0043ache.data
ClusterNode-node1
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2
ContainerBackgroundProcessor[StandardEngine[Catalina]]
DynamicPooledExecutor
Finalizer
http-bio-0.0.0.0-4080-Acceptor-0
http-bio-0.0.0.0-4080-AsyncTimeout
http-bio-0.0.0.0-4080-exec-109
http-bio-0.0.0.0-4080-exec-114
http-bio-0.0.0.0-4080-exec-120
http-bio-0.0.0.0-4080-exec-121
http-bio-0.0.0.0-4080-exec-124
http-bio-0.0.0.0-4080-exec-125
http-bio-0.0.0.0-4080-exec-26
http-bio-0.0.0.0-4080-exec-50
http-bio-0.0.0.0-4080-exec-51
http-bio-0.0.0.0-4080-exec-53
jackrabbit-pool-1
jackrabbit-pool-10
jackrabbit-pool-11
jackrabbit-pool-12
jackrabbit-pool-13
jackrabbit-pool-14
jackrabbit-pool-15
jackrabbit-pool-16
jackrabbit-pool-2
jackrabbit-pool-3
jackrabbit-pool-4
jackrabbit-pool-5
jackrabbit-pool-6
jackrabbit-pool-7
jackrabbit-pool-8
jackrabbit-pool-9
Java2D Disposer
main
MultiThreadedHttpConnectionManager cleanup
net.sf.ehcache.CacheManager@15aec0
ObservationManager
ObservationManager
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-10
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-11
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-12
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-13
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-14
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-15
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-16
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-17
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-18
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-19
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-20
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-21
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-22
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-23
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-24
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-25
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-4
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-5
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-6
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-7
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-9
process reaper
process reaper
process reaper
process reaper
QuartzScheduler_org.springframework.scheduling.quartz.SchedulerFactoryBean#0-bf1-rmaicic1.ads.bf1.yahoo.com1370389426455_ClusterManager
QuartzScheduler_org.springframework.scheduling.quartz.SchedulerFactoryBean#0-bf1-rmaicic1.ads.bf1.yahoo.com1370389426455_MisfireHandler
Reference Handler
Signal Dispatcher
Timer-0
Timer-1
Timer-2
Timer-3
Timer-4
