This might be useless, but if you have nothing else to try, you can try using the ip address instead of machine name in the jdbc url. On Jun 11, 2015 12:51, "Romain Manni-Bucau" <[email protected]> wrote:
> Le 11 juin 2015 09:49, "Mark Struberg" <[email protected]> a écrit : > > > > Hmm, if there is a network issue then the connection should actually get > closed, right? > > Or not depends the setup but network but maybe not the right word. A > proxy/firewall can hold it for weeks even if behind there is nore any > instance/db. > > > So the lock gets cleaned and you will see an Exception in your app. > > > > Sounds really like a weird issue and I did not yet see this on any system > yet. The more important it is to get behind the real reason why it stales > for you. > > > > Let’s try to dissect this from top-down. > > > > You have 507 threads overall: > > ~/tmp/delete/log$>grep java.lang.Thread.State log1.txt | wc -l > > 507 > > > > Almost all of them are in a waiting state: > > ~/tmp/delete/log$>grep "java.lang.Thread.State: TIMED_WAITING" log1.txt > |wc -l > > 459 > > > > > > There are 64 connections waiting for the database, right? > > > > ~/tmp/delete/log$>grep > > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection\(ConnectionPool.java:650 > log1.txt |wc -l > > 64 > > > > > > We have quite a few EJBs waiting on a lock: > > ~/tmp/delete/log$>grep org.apache.openejb.util.Pool.pop\(Pool.java:224 > log1.txt | wc -l > > 139 > > But that are by far not all threads. > > > > We also have 5 open sockets waiting for a connection: > > ~/tmp/delete/log$>grep > "java.net.ServerSocket.accept.ServerSocket.java:513“ log1.txt | wc -l > > 5 > > > > Tons of small fishes, like 28 hazelcast threads, etc… > > > > And one ‚big fish‘ > > > > ~/tmp/delete/log$>grep org.apache.openejb.pool.scheduler log1.txt | wc -l > > 233 > > That confuses me a bit to be honest. > > > > LieGrue, > > strub > > > > > > > Am 10.06.2015 um 20:21 schrieb Romain Manni-Bucau < > [email protected] > >: > > > > > > 2015-06-10 11:17 GMT-07:00 Matej <[email protected]>: > > > > > >> Hi. > > >> > > >> Maybe what i forgot to tell is that when looking at the database we > see > > >> many connections like that: > > >> > > >> IDLE IN TRANSACTION, mayn (ALL) transaction that are long running, > waiting > > >> for commit statemet which somehow doesn't happen. > > >> > > >> And the CPU load on DB and App server is almost 0, so the system does > seem > > >> to lock up. > > >> > > >> Romain, last question can 2 or more transaction share the same > connection, > > >> maybe this could cause some locks to deadlock?.... > > >> > > >> > > > shouldnt normally > > > > > > Now on what you observe it really looks like the connection is somehow > lost > > > on client side. Maybe try to kill connection on DB side, it could throw > an > > > error on application side and unlock the app maybe. If so the network > has > > > an issue somewhere. > > > > > > > > >> > > >> BR > > >> > > >> Matej > > >> > > >> > > >> > > >> > > >> 2015-06-10 20:01 GMT+02:00 Romain Manni-Bucau <[email protected] > >: > > >> > > >>> 2015-06-10 10:59 GMT-07:00 Matej <[email protected]>: > > >>> > > >>>> Hi Romain. > > >>>> > > >>>> We lookef for DB locks, but it's only SELECT and INSERT statements, > > >> which > > >>>> should not cause locking on postgresql. And we also looked for locks > > >>> using > > >>>> postgresql debuging, analytics. So we suspected the network and the > > >> JDBC > > >>>> driver, we did some TCP stack tuning, but also no major change. > > >>>> > > >>>> > > >>> was more thinking about locks in the datasource pool than on db side > (ie > > >>> pool size too small) > > >>> > > >>> > > >>>> Everything does run on VMWARE, so maybe this could be also something > to > > >>>> look at. We could add some more Tomcat DB pool timeouts, but this is > a > > >>>> dirty "fix". > > >>>> > > >>> > > >>> set the timeout to maybe 0 to set the app failing is there is not > enough > > >>> connections in the pool > > >>> > > >>> > > >>>> > > >>>> Can this be a phenomen of not enough connections in pool. But I > would > > >>>> suspect a performance drop not lock. > > >>>> > > >>> > > >>> well i dont see locks in the logs but "locked" which means waiting > for > > >>> something to happen (getting the connection) > > >>> > > >>> > > >>>> > > >>>> Can somebody explain this: is 1 transaction == 1 connection. So > > >>> connected > > >>>> beans share the same connection and transcation? > > >>>> > > >>> > > >>> by datasource you have 1 connection per transaction yes > > >>> > > >>> > > >>>> > > >>>> BR > > >>>> > > >>>> Matej > > >>>> > > >>>> 2015-06-10 19:24 GMT+02:00 Romain Manni-Bucau < > [email protected] > >: > > >>>> > > >>>>> well before playing with config the idea is to know what happened, > is > > >>> the > > >>>>> network quality the cause for instance? > > >>>>> > > >>>>> > > >>>>> Romain Manni-Bucau > > >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog > > >>>>> <http://rmannibucau.wordpress.com> | Github < > > >>>>> https://github.com/rmannibucau> | > > >>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber > > >>>>> <http://www.tomitribe.com> > > >>>>> > > >>>>> 2015-06-10 10:22 GMT-07:00 Howard W. Smith, Jr. < > > >>> [email protected] > > >>>>> : > > >>>>> > > >>>>>> On Wed, Jun 10, 2015 at 1:18 PM, Romain Manni-Bucau < > > >>>>> [email protected] > > >>>>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Hi Matej > > >>>>>>> > > >>>>>>> looks like a database issue for me (stateless pool is waiting for > > >>> the > > >>>>>>> stateless trying to get the db connection to be released), hasnt > > >>> the > > >>>> DB > > >>>>>>> connection be lost or something like that? > > >>>>>>> > > >>>>>>> > > >>>>>> what is the best way for Matej to improve that? tomcat jdbc pool > > >>>>> settings? > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > >
