Hello,
We are running tomcat 6.0.36 (java build 1.6.0_31) on REHL5. It works most
of time except recently we started running out of files. Here is the error
I am getting:
Exception in thread "pool-1-thread-1" java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "pool-1-thread-2" java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:406)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:18:55 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
access:2013-06-22 20:18:50.645
Jun 22, 2013 8:19:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
access:2013-06-22 20:18:55.649
Jun 22, 2013 8:19:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@378186dc last
access:2013-06-22 20:18:55.646
Jun 22, 2013 8:19:05 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@6ec71370 last
access:2013-06-22 20:19:00.648
Jun 22, 2013 8:19:05 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
WARNING: Channel key is registered, but has had no interest ops for the
last 3000 ms. (cancelled:false):sun.nio.ch.SelectionKeyImpl@71312bbd last
access:2013-06-22 20:19:00.649
SEVERE: Unable to process request in NioReceiver
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:758)
at net.bull.javamelody.Action.stopThread(Action.java:388)
at net.bull.javamelody.Action.killThread(Action.java:374)
at net.bull.javamelody.Action.execute(Action.java:227)
at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:19:55 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
SEVERE: Unexpected poller error
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:758)
at net.bull.javamelody.Action.stopThread(Action.java:388)
at net.bull.javamelody.Action.killThread(Action.java:374)
at net.bull.javamelody.Action.execute(Action.java:227)
at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:20:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
SEVERE: Unexpected poller error
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:758)
at net.bull.javamelody.Action.stopThread(Action.java:388)
at net.bull.javamelody.Action.killThread(Action.java:374)
at net.bull.javamelody.Action.execute(Action.java:227)
at
net.bull.javamelody.MonitoringController.executeActionIfNeeded(MonitoringController.java:117)
at
net.bull.javamelody.MonitoringController.doActionIfNeededAndReport(MonitoringController.java:130)
at
net.bull.javamelody.MonitoringFilter.doMonitoring(MonitoringFilter.java:340)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:227)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:448)
at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:403)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1703)
at java.lang.Thread.run(Thread.java:662)
Jun 22, 2013 8:20:00 PM
org.apache.catalina.tribes.transport.nio.NioReceiver socketTimeouts
Then I get a bunch of
Exception in thread
"org%002ejasig%002ecas%002eticket%002e%0054icket%0047ranting%0054icket.data"
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:127)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1239)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:176)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Eventually, the filesystem fills up and tomcat is dead. Any ideas of why
we are getting the thread errors. Thanks in advance!
tom
--
*****************************
Tom Freestone
(tefreest...@gmail.com)
*****************************
