On Thu, Sep 27, 2018 at 3:56 AM Louis Zipes <louis.zi...@gmcr.com> wrote:
> Problem just re-occurred and so I was able to at least get a JSTACK (I > assume it was Tomcat since it was the Java using the most memory on the > machine). Here is the reoccurring message. I get more hits on but haven't > dug through all of the Google hits yet (due to multi-tasking) so apologies > up front if there is a simple answer to this. > > "Event_Manager_1413" daemon prio=6 tid=0x0000000024856000 nid=0x40c4 > waiting on condition [0x0000000042dae000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000005ab45f7b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown > Source) > at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > > Do you lock from memory or database? Make sure locking sequence is consistent all across the application. Meaning if you have Lock A Lock B Lock C You dont want another logic that locks something in another order. E.g. Lock B Lock A Lock C Because that will cause deadlocks. > Locked ownable synchronizers: > - None > > > Any comments/suggestions are appreciated! > > Your most likely problem is database connection pool mismanagement: > connections aren't properly released and the pool empties. All threads > are left waiting on available database connections which will never be > replenished. > > I'm using the ojdbc6.jar if that is what you are referring to or is there > a better setting somewhere. > What db connection pool are you using? Are you letting tomcat manage it via Context.xml and get connection via JNDI? Or are you managing inside the application? > > -----Original Message----- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Wednesday, September 26, 2018 3:46 PM > To: users@tomcat.apache.org > Subject: Re: Application hanging on Tomcat 7.0.54 > > - - - external message, proceed with caution - - - > > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Louis, > > On 9/26/18 14:42, Louis Zipes wrote: > > Hi all, Tomcat 7.0.54 running on Windows 2012 > > > > We are running a third party application on Tomcat and today we > > have intermittently run in issues where the application stops > > working. The big changes in our system is that we have added more > > end users and we are at year end so of course everyone is hitting > > the system hard. Even if we force a log out of all users and stop > > all background jobs then the application doesn't recover. > > > > We see no active sessions on the database (our application is > > connecting to an Oracle database) and I see no clear error messages > > in either our third party application logs or the Tomcat logs (ex. > > OutofMemory). When we go to the Windows Task Manager we did not > > see the machine's Memory max'd out but admittedly I didn't look at > > the Java session to see if was reaching its Heap Max. The only > > thing that we noticed was that TCP connections went down right > > after the restart. I did open up Jconsole under Java and I did > > force a garbage collection but that didn't seem to help. > > > > We do have an Oracle Grid Control and we did get an alert in > > regards to Metric: [HTTP Transaction] Perceived Time per Page going > > past thresholds but not sure if that was just an old alert with and > > old range that was set up a long time ago or is a really valid > > clue. Since this is PRD we had to get it back up and running so > > all I did was increase the Tomcat Xmx Heap size and restarted. I'm > > not really confident that is the solution since as mentioned you > > tend to see a clear out of memory error if it was too small. > > > > So a few questions: > > > > > > 1) Does this sound like a known issue with this earlier version > > of Tomcat? > > No. > > > 2) Should I turn up any logging on Tomcat and if so which > > ones? > > Not yet. > > > 3) We didn't do a JSTACK dump while it was happening. Would > > that have been useful? > > Absolutely. > > > 4) Do we need to play around with MaxThreads and/or > > MaxConnections. We do have maxThreads in our server.mxl but in DEV > > when we turned it down to a value = 5 hoping to overwhelm it > > nothing bad happened. > > Don't change anything, yet. > > > Once again, we are limited to what we could do and collect since it > > was PRD and we needed to restart it. We restarted the Tomcat > > service and everything is processing fine for right now. I will > > note that that we did have that bad Windows patch that prevented it > > from stopping and starting cleanly > > (https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdow > n) > > but we have taken the break fix patch and the daily restarts seem > > to be fine since then. > > > > Any comments/suggestions are appreciated! > > Your most likely problem is database connection pool mismanagement: > connections aren't properly released and the pool empties. All threads > are left waiting on available database connections which will never be > replenished. > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlur4fgACgkQHPApP6U8 > pFhq/RAAwZixHqbHxYdX3VCrTfvz0tnOmu7W4sbeFqhExV+M6NVL2LK1RO26eTq5 > OJB2o/RheCeajWHxqiJQY4ERMTOyyqZYCsRG9L901heW2UAW122zeX7hhXDB1IMo > qIBVYQalg1j5e2Lw9MqT3ISj6U/GNA6VlirTAHGtuEBBpKXXyb6KtmOgpjHjjXS7 > mTYni2iTHO/NaGrS519alFPMBnF4Wq5NzRcLewNMqj9Nbx2uu3Suu95DhJ+WIIep > DOmyy4UGHxGx2QqNUnVWMHApGGjFD4pTWIBwzcbbsL56kZDxRGsF0SsB0VR/jSMY > HdI71RgjpQFSyar1rcCTPCRP97KnUC+oaKhn+i2jBMSRxs94GEOqk4LuKNo39HKP > tXEMkYl0o/CR9QaFDjncy9M2M3/o50ooBvYTOu0SjmyZO+ab9tJpaXbnlf2ChxJI > AWbhBGmwJ6kF5FvVbmzujV7EEF2YCRMBpWo2zjNd6zWvX9OWEOXrOaHuX7iBPCga > YqEMQWyS7XWiBG7AI8+ka5x4s/oxWsbn/6pCdDXhfxl5p5jv7ajm7LbLXGut1N6a > uV5hmLLgZywF68AYe6X3GWv6mygXMBYABZxEA6klWE7HpIzvLxmxu0+vFlYR8qsb > FMAacJ/FYJ9nGRuMqG+V2Edr5U//JvrWqy4raPIvwXGtX+FsqEc= > =Zavb > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------- > CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and > may contain information that is confidential, proprietary or exempt from > disclosure. If you are not the intended recipient, please contact the > sender immediately. Unauthorized use or distribution is prohibited and may > be unlawful. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Guang <http://javadevnotes.com/java-string-to-char-array>
