Sounds like you have a thread holding a lock on a critical table and subsequent threads are lining up behind it waiting for it to finish. You should check your MySQL to try and figure out who's holding the lock and why.
Note that the culprit thread need not be hung up in the database. Locks are held until the transaction commits, so your thread could be hung up in app code after making a database update but before the transaction is committed. jk On Wed, Apr 15, 2009 at 03:41:07PM +0200, J wrote: > (Sorry for the empty message. First I tried Gmail, but that doesn't work > with this mailing list. Then I tried GMX webclient, but that client always > sends as HTML, which probably caused the message to be stripped to empty.) > > > I'm experiencing freezes on a production website. > > Server specs: > -OS: Linux CentOS > -Webserver: Tomcat 6 > -MySQL 5 with default settings > -Frameworks: Wicket 1.4-rc2 (webframework), Spring 2.5.6, Hibernate > 3.3.1.GA, C3P0 db pooling 0.9.1.2, URLRewriteFilter 3.1.0, Spring > OpenSessionInViewFilter > > Problem: > At some point (about 5 to 24 hours) after a boot, Tomcat seems to stop > serving requests, although it does create new threads for new incoming > connections. But since it does not serve those connections, the new > connections will show and keep showing 0kb transfer. Shortly after that, > max-thread is reached, so then I'm unable to access any webapp (the main > website + tomcat manager) running on that Tomcat server. I'm not sure if it > is able to serve static resources in the short time window where it still > has some threads free, because the time window is too short to notice when > it happens. > > Observations: > -No exceptions or errors in the catalina logs. So no memory problems, since > no error occurs in the logs > -Java Thread dump using command: kill -QUIT (see output below), shows the > text "locked" and "WAITING". > -Adding c3p0 db pooling idle checks, tests and timeout settings (see below) > did not help. > -Using Mysql Administrator GUI shows that after a freeze, there will be 15 > threads, all sleeping. Normally this would return to a minPoolSize of 5, if > I'm correct, but thats not the case. > -The freeze continues for hours, and does not recover. I have to restart > Tomcat. > > > I'm suspecting that it has something to do with DB pooling. I'm not sure if > "locked" and "WAITING" are normal behaviour. But if there is something > wrong, why doesn't c3p0 recover from it? > > Can somebody shed some light on this? :) > > > > > > Configuration: > > === DB pooling in Spring applicationContext.xml ==== > <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"> > <property > name="driverClass"><value>com.mysql.jdbc.Driver</value></property> > <property name="jdbcUrl" value="jdbc:mysql://localhost/xxxxxx" /> > <property name="user"><value>xxxxxxxx</value></property> > <property name="password"><value>xxxxxxxxx</value></property> > > <property name="maxConnectionAge"><value>600</value></property> > <property > name="idleConnectionTestPeriod"><value>180</value></property> > <property > name="testConnectionOnCheckin"><value>true</value></property> > > <property name="acquireIncrement"><value>5</value></property> > <property name="maxIdleTime"><value>180</value></property> > <property name="maxPoolSize"><value>15</value></property> > <property name="maxStatements"><value>100</value></property> > <property name="minPoolSize"><value>5</value></property> > </bean> > =========================== > > > ===== Thread dump part =========== > "http-8080-28" daemon prio=10 tid=0xb4fbb400 nid=0x4ae2 in Object.wait() > [0xb4316000..0xb4318130] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePo > ol.java:1315) > at > com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicRe > sourcePool.java:557) > - locked <0x72c74b10> (a > com.mchange.v2.resourcepool.BasicResourcePool) > at > com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResource > Pool.java:477) > at > com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C > 3P0PooledConnectionPool.java:525) > at > com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstract > PoolBackedDataSource.java:128) > at > org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConn > ection(LocalDataSourceConnectionProvider.java:81) > at > org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:4 > 23) > at > org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:14 > 4) > at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119) > at > org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57) > at > org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326) > at > org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber > nateTransactionManager.java:510) > at > org.springframework.transaction.support.AbstractPlatformTransactionManager.g > etTransaction(AbstractPlatformTransactionManager.java:350) > at > org.springframework.transaction.interceptor.TransactionAspectSupport.createT > ransactionIfNecessary(TransactionAspectSupport.java:262) > at > org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr > ansactionInterceptor.java:101) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect > iveMethodInvocation.java:171) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro > xy.java:204) > at $Proxy37.findById(Unknown Source) > at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.wicket.proxy.LazyInitProxyFactory$JdkHandler.invoke(LazyInitProxy > Factory.java:416) > at org.apache.wicket.proxy.$Proxy2.findById(Unknown Source) > at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > at > org.apache.wicket.model.LoadableDetachableModel.getObject(LoadableDetachable > Model.java:119) > at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc > torAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java > :151) > at > org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java > :89) > at > org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.new > Page(BookmarkablePageRequestTarget.java:306) > at > org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.get > Page(BookmarkablePageRequestTarget.java:321) > at > org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.pro > cessEvents(BookmarkablePageRequestTarget.java:234) > at > org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(Abstra > ctRequestCycleProcessor.java:92) > at > org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:119 > 2) > at org.apache.wicket.RequestCycle.step(RequestCycle.java:1271) > at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1370) > at org.apache.wicket.RequestCycle.request(RequestCycle.java:501) > at > org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:455) > at > org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:288) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application > FilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh > ain.java:206) > at > org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterI > nternal(OpenSessionInViewFilter.java:198) > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF > ilter.java:76) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application > FilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh > ain.java:206) > at > org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164 > ) > at > org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141) > at > org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.jav > a:90) > at > org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter > .java:417) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application > FilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh > ain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja > va:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja > va:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128 > ) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102 > ) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java > :109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http > 11Protocol.java:583) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) > at java.lang.Thread.run(Thread.java:619) > ======================================= > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org > For additional commands, e-mail: users-h...@wicket.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org For additional commands, e-mail: users-h...@wicket.apache.org
