On Sat, Nov 22, 2014 at 4:03 PM, Dhaval Jaiswal <dhaval.jais...@via.com> wrote:
> Is your postgresql server responsive? > Yes, there is no error logged in db server and server is very much > responsive. > > > Have you ever been able to get a valid connection during the lifetime of > the application? > Yes > > Was it working for sometime, and then you suddenly start getting this > issue...? > Yes. it happenes suddendly and sometime not always. > > Assuming you have isolated this thread correctly as one which is stuck(by > taking multiple thread dump...) for how long it has been stuck? > > Many thread dump took, analyzed and says thread stuck while getting the > connection from the db or trying to execute queries. > Is it possible firewall dropping connection when Apps server trying to get > the db connection. > > I don't think, firewall is an issue, as you said you are able to get connection successfully for sometime and this is happening only intermittently. > > Is it ever coming out of that stuck state? > yes. However, number of threads are not coming down. > How is it coming out? Does it throws SocketTimeOutException or does it return back normally returning a valid connection. But in any case, your thread is no lunger stuck. If number of thread is not coming down, it is because it must be busy doing other things. You can't expect the thread to die down just after getting a connection. It must proceed on doing other stuff in your code flow. > > Look into the state of tcp connection using netstat. > 1 FIN_WAIT1 > 1 Foreign > 1 established) > 2 LAST_ACK > 2 SYN_SENT > 4 CLOSING > 10 LISTEN > 17 CLOSE_WAIT > 116 TIME_WAIT > 287 ESTABLISHED > > You can use below command to get the state of connection opened by your tomcat process: netstat -atp|grep <pid of the tomcat process> In any case, by the symptom you mentioned(connection hunged intermittently),I highly expect it is the postgresql server which is causing the issue. You should check the load at that specific time at your db server. Try connecting an alternate, probably back db server, to see is issue still exist, which I'm to certain extent sure, it will not. > > > > On Sat, Nov 22, 2014 at 3:08 PM, Ameer Mawia <ameer.ma...@gmail.com> > wrote: > > > Jaiswal, > > > > This issue is of connection between your host and postgresql server, you > > are trying to connect and has nothing to do with Tomcat per say. > > > > To debug the issue, you will have to answer few more question: > > Is your postgresql server responsive? > > Have you ever been able to get a valid connection during the lifetime of > > the application? Was it working for sometime, and then you suddenly start > > getting this issue...? > > Assuming you have isolated this thread correctly as one which is stuck(by > > taking multiple thread dump...) for how long it has been stuck? Is it > ever > > coming out of that stuck state? > > Look into the state of tcp connection using netstat. > > > > Regards, > > Ameer Mawia > > > > > > On Sat, Nov 22, 2014 at 10:44 AM, Dhaval Jaiswal <dhaval.jais...@via.com > > > > wrote: > > > > > yes. the thread is stucked and started increasing load of the server. > > > > > > My configuration is as below. Query is fetching one row only. > > > > > > > > > <Resource name="jdbc" auth="Container" > > type="javax.sql.DataSource" > > > driverClassName="org.postgresql.Driver" url="jdbc:postgresql" > > > maxActive="100" maxIdle="20" maxWait="30000" > > > validationQuery="select 1" testOnBorrow="true" > > > removeAbandoned="true" removeAbandonedTimeout="120" > > > logAbandoned="true" /> > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 22, 2014 at 1:29 AM, Ameer Mawia <ameer.ma...@gmail.com> > > > wrote: > > > > > > > Jaiswal, what your exact question is? > > > > > > > > Obviously in your application, you need a DB connection and if this > is > > > > thread which is stuck, stack trace tells that tomcat dbcp, is trying > to > > > get > > > > you a connection, but stuck while reading the result of validate > query > > > > which it had run before returning you a valid connection. > > > > > > > > Probably you can take a look at validation query, for now? Ideally it > > > > should be a very simple query, returning just one row and column. > > > > > > > > Regards, > > > > Ameer Mawia > > > > > > > > On Sat, Nov 22, 2014 at 12:30 AM, Dhaval Jaiswal < > > dhaval.jais...@via.com > > > > > > > > wrote: > > > > > > > > > Following thread stuck and increased the load. > > > > > > > > > > > > > > > java.net.SocketInputStream.read(SocketInputStream.java:129) > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145) > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) > > > > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274) > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660) > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374) > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > Source) > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > Source) > > > > > > > > > > > > > > > > > > > > com.eos.accounts.policies.PolicyManager.getAllPoliciesForUser(PolicyManager.java:62) > > > > > > > > > > > > > > > > > > > > com.eos.accounts.policies.Policies.loadAllPoliciesForUser(Policies.java:25) > > > > > com.eos.accounts.data.User.createFromDataSet(User.java:3165) > > > > > com.eos.accounts.data.User.getUserById(User.java:2878) > > > > > > > > > > > > > > > > > > > > com.eos.b2c.beans.GenericProductBean.setDeliveryDetails(GenericProductBean.java:320) > > > > > > > com.eos.b2c.ui.ResellerNavigation.doGet(ResellerNavigation.java:2362) > > > > > > > com.eos.b2c.ui.ResellerNavigation.doPost(ResellerNavigation.java:579) > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:710) > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > > > > > > > > > > > > > > > > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > > > > > > > > > > > > > > > > > > > thread dump > > > > > > > > > > > > > > > "http-8080-2" daemon prio=10 tid=0x000000005335e000 nid=0x6d82 > > runnable > > > > > [0x00002b8b9a5c9000..0x00002b8b9a5caa90] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at > > java.net.SocketInputStream.read(SocketInputStream.java:129) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) > > > > > at > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) > > > > > - locked <0x00002b881d689760> (a > > > > > org.postgresql.core.v3.QueryExecutorImpl) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374) > > > > > at > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > > > > > at > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > Source) > > > > > at > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > Source) > > > > > at > > > > > > > > > > > > > > > > > > > > com.eos.gds.util.GenericSearchQuery.executeQuery(GenericSearchQuery.java:191) > > > > > > > > > > > > > > > > > > > > On Sat, Nov 22, 2014 at 12:26 AM, Dhaval Jaiswal < > > > dhaval.jais...@via.com > > > > > > > > > > wrote: > > > > > > > > > > > Below thread stucked and increased the load. > > > > > > > > > > > > > > > > > > java.net.SocketInputStream.read(SocketInputStream.java:129) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) > > > > > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > > > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > > Source) > > > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > > Source) > > > > > > > > > > > > > > > > > > > > > > > > > > > com.eos.accounts.policies.PolicyManager.getAllPoliciesForUser(PolicyManager.java:62) > > > > > > > > > > > > > > > > > > > > > > > > > > > com.eos.accounts.policies.Policies.loadAllPoliciesForUser(Policies.java:25) > > > > > > com.eos.accounts.data.User.createFromDataSet(User.java:3165) > > > > > > com.eos.accounts.data.User.getUserById(User.java:2878) > > > > > > > > > > > > > > > > > > > > > > > > > > > com.eos.b2c.beans.GenericProductBean.setDeliveryDetails(GenericProductBean.java:320) > > > > > > > > > com.eos.b2c.ui.ResellerNavigation.doGet(ResellerNavigation.java:2362) > > > > > > > > > com.eos.b2c.ui.ResellerNavigation.doPost(ResellerNavigation.java:579) > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:710) > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > > > > > > > > > > > > > > > > > > > > > > > thread dump output > > > > > > > > > > > > > > > > > > "http-8080-2" daemon prio=10 tid=0x000000005335e000 nid=0x6d82 > > > runnable > > > > > > [0x00002b8b9a5c9000..0x00002b8b9a5caa90] > > > > > > java.lang.Thread.State: RUNNABLE > > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > > at > > > java.net.SocketInputStream.read(SocketInputStream.java:129) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) > > > > > > at > > > org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) > > > > > > - locked <0x00002b881d689760> (a > > > > > > org.postgresql.core.v3.QueryExecutorImpl) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:991) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > > > > > > at > > > > > > > > > > > > > > > > > > > > > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > > > > > > at > > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > Source) > > > > > > at > > > > > > > com.eos.accounts.util.AccountsDatabaseManager.getConnection(Unknown > > > > > Source) > > > > > > at > > > > > > > > > > > > > > > > > > > > > com.eos.gds.util.GenericSearchQuery.executeQuery(GenericSearchQuery.java:191) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
