Am 31.08.20 um 06:26 schrieb Gokhan Akgul: > Dear Felix, > > Thanks for your feedback , the health check period is so aggressive now. > SRE team's and apm monitor tools call internally and externally endpoint > and health indicators. > I asked the sre guys to decrease the frequency of those calls. They made > some decrease on calls but nothing changed. No, don't decrease the frequency. I meant to have the calls happen more often than every ten minutes. > Due to an old spring mvc project it doesn't have a kubernetes probe , about > 100 calls in minutes.
Those calls have to use the connection, that the health check is using. Maybe you could check, whether the healtcheck is releasing its connection back to the pool after it has done its work. Felix > > Gökhan > > On Thu, Aug 27, 2020 at 9:48 PM Felix Schumacher < > felix.schumac...@internetallee.de> wrote: > >> Am 27.08.20 um 11:47 schrieb Gokhan Akgul: >>> Hi , >>> >>> I have been facing the deadlock issue for the last 2 months about >>> JDBCPoolCleaner Thread . >>> >>> Following config set in context.xml >>> >>> <Resource name="jdbc/database" >>> auth="Container" >>> type="javax.sql.DataSource" >>> factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" >>> driverClassName="com.mysql.jdbc.Driver" >>> >> url="jdbc:mysql://adress:3306/db?useUnicode=true&characterEncoding=latin5&characterResultSet=latin5&zeroDateTimeBehavior=convertToNull&autoReconnect=true&interactiveClient=true" >>> username="user" >>> password="pass" >>> initialSize="10" >>> maxActive="30" >>> maxIdle="15" >>> minIdle="10" >>> maxWait="30000" >>> timeBetweenEvictionRunsMillis="5000" >>> minEvictableIdleTimeMillis="60000" >>> removeAbandonedTimeout="600" >>> removeAbandoned="true" >>> logAbandoned="false" >>> testWhileIdle="true" >>> testOnBorrow="true" >>> testOnReturn="false" >>> validationQuery="/* ping */ SELECT 1" >>> validationInterval="30000" >>> jmxEnabled="true" >>> >> jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTimer;SlowQueryReport" >>> /> >>> >>> >>> >>> Thread dump >>> >>> Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 state=BLOCKED >>> - waiting to lock <0x57dcb0b7> (a >> com.mysql.jdbc.JDBC4PreparedStatement) >>> owned by http-nio-8080-exec-8 id=25 >> So, Tomcat tries to close a connection, that it thinks is abandoned >> (i.e. in your setup hasn't been used for more than 600 seconds and the >> pool of idle connections is getting empty). >> >> The connection is still in use ... >> >>> at >> com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078) >>> at >> com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) >>> at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) >>> at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) >>> at >> org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:388) >>> at >> org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.java:618) >>> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java:612) >>> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:569) >>> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPool.java:999) >>> at >> org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1468) >>> at java.util.TimerThread.mainLoop(Timer.java:555) >>> at java.util.TimerThread.run(Timer.java:505) >>> >>> Locked synchronizers: count = 1 >>> - >> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868 >>> >>> >>> http-nio-8080-exec-8 id=25 state=BLOCKED >>> - waiting to lock <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) >>> owned by Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16 >> ... by hibernate. The question now is, how often is your healtheck >> getting called (every 10 min == 600s)? >> >> If they are close together, you might want to set the abandoned timeout >> higher than the healthcheck interval, or you could try to close the >> connection (or whatever the equivalent is in hibernate (a session?)) in >> your code. >> >> Felix >> >>> at >> com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.java:5435) >>> at >> com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaData.java:3269) >>> at com.mysql.jdbc.DatabaseMetaData.<init>(DatabaseMetaData.java:675) >>> at >> com.mysql.jdbc.JDBC4DatabaseMetaData.<init>(JDBC4DatabaseMetaData.java:39) >>> at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown >> Source) >>> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>> at >> com.mysql.jdbc.DatabaseMetaData.getInstance(DatabaseMetaData.java:657) >>> at >> com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3064) >>> at >> com.mysql.jdbc.ConnectionImpl.getMetaData(ConnectionImpl.java:3056) >>> at >> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2315) >>> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) >>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >> org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) >>> at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) >>> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) >>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >> org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) >>> at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) >>> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) >>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >> org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) >>> at com.sun.proxy.$Proxy44.executeQuery(Unknown Source) >>> at >> org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208) >>> at org.hibernate.loader.Loader.getResultSet(Loader.java:1953) >>> at org.hibernate.loader.Loader.doQuery(Loader.java:802) >>> at >> org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274) >>> at org.hibernate.loader.Loader.doList(Loader.java:2542) >>> at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276) >>> at org.hibernate.loader.Loader.list(Loader.java:2271) >>> at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459) >>> at >> org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365) >>> at >> org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) >>> at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268) >>> at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) >>> at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:246) >>> >>> ...... >>> >>> at >> com.gg.healthcheck.servlet.HealthCheckServlet.getServiceStatus(HealthCheckServlet.java:68) >>> at >> com.gg.healthcheck.servlet.HealthCheckServlet.doGet(HealthCheckServlet.java:45) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) >>> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) >>> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >>> at >> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) >>> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) >>> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >>> at >> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) >>> at >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) >>> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) >>> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) >>> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) >>> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) >>> at >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) >>> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) >>> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) >>> at >> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) >>> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) >>> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) >>> at >> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615) >>> at >> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) >>> at >> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) >>> at org.apache.tomcat.util.net >> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626) >>> at org.apache.tomcat.util.net >> .SocketProcessorBase.run(SocketProcessorBase.java:49) >>> - locked org.apache.tomcat.util.net >> .NioEndpoint$NioSocketWrapper@526aa725 >>> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at >> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Locked synchronizers: count = 1 >>> - java.util.concurrent.ThreadPoolExecutor$Worker@21c6b5b9 >>> >>> >>> >>> >>> >>> Tomcat version : 8.8.57 >>> Java 7 >>> Application is a simple spring mvc application. >>> Runs on Kubernetes >>> >>> >>> >>> Do you have any idea how to solve it? >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
