Hi Thomas,
> -----Original Message----- > From: Thomas Hoffmann (Speed4Trade GmbH) > <thomas.hoffm...@speed4trade.com.INVALID> > Sent: Sunday, May 26, 2024 3:30 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: AW: Database Connection Requests Initiated but Not Sent on the Wire > (Some, Not All) > > Hello, > > > -----Ursprüngliche Nachricht----- > > Von: Chuck Caldarale <n82...@gmail.com> > > Gesendet: Sonntag, 26. Mai 2024 21:21 > > An: Tomcat Users List <users@tomcat.apache.org> > > Betreff: Re: Database Connection Requests Initiated but Not Sent on > > the Wire (Some, Not All) > > > > > > > On May 25, 2024, at 20:58, Eric Robinson <eric.robin...@psmnv.com> > wrote: > > > > > > One of our hosting customers is a medical practice using a > > > commercial EMR > > running on tomcat+mysql. It has operated well for over a year, but > > users have suddenly begun experiencing slowness for about an hour at > > the same time every day. During the slow times, we've done all the > > usual troubleshooting to catch the problem in the act. The servers > > have plenty of power and are not overworked. There are no slow database > queries. Network connectivity is solid. > > Tomcat has plenty of memory. The numbers of database connections, > > threads, questions, queries, etc., remain steady, without spikes. > > There is no unusual disk latency. We have not found any maintenance > > tasks running during that timeframe. > > > > > > <snip/> > > > > > > > There are no unusual errors in the tomcat or database server logs, > > > EXCEPT > > this one: Java.sql.DriverManager.getConnection > > > > > > <snip/> > > > > > > > During the periods of slowness, we see lots of those errors along > > > with a large > > spike in the number of stuck tomcat threads (from 1 or 2 to as high as > > 100). It seems obvious that the threads are stuck because tomcat is > > waiting on a connection to the database. > > > > > > <snip/> > > > > > > > We are forced to conclude that some database connection requests are > > > being > > initiated but are not being sent on the wire. > > > > > > Could the DB server be out of ports? (Seems unlikely, based on your > > debugging so far.) > > > > Any chance that the Tomcat process is running out of file descriptors? Or > ports? > > > > Can you force a garbage collection (e.g., with jconsole or similar > > tool) during a slow period? If there is some limit on an OS-level > > resource that’s being reached, a GC may be able to delete the Java > > objects that are tying up the underlying resources. > > > > - Chuck > > > > > On the client side, the TCP connections are kept in a wait-state for usually 2 > minutes as far as I know. > Maybe you can check how many are in this state. > On our server, we set things much lower to allow faster recycling of TCP connections... net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_tw_reuse = 1 > If the application doesn’t use connection pooling, then this can be the > problem > itself too. During peak production, there are a total of around 20,000 connections in various states, mostly TIME_WAIT. The port range is 5000-60000. Dmesg, journalcrl, and the messages file don't show any errors about running out of ports or file handles. > TCP handshakes and logon process take a while and for performance reasons, > DB connections are usually pooled. > > A stacktrace might help to see what java is doing when it enters this blocking > state. > Maybe you can provide a stack when the app starts blocking. > We are writing a script to watch for stuck threads to exceed a threshold, and do a thread dump when that happens. > Greetings, > Thomas Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
