Alan,
> -----Original Message----- > From: Alan F <shiva...@hotmail.com> > Sent: Thursday, February 03, 2022 2:51 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: RE: help with high cpu usage > > My bad here are the correct dumps 10 secs apart > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0zNzY2ZmZmNy0wZDgyL > TRhZTItYmE3Mi0zMWQyYTYwN2M1ZjgudHh0&__;!!F9svGWnIaVPGSwU!- > fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG > st67ghaaLsg$ > > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS00NTE3MWUxNy1jYWRi > LTRkY2UtODBlNS1lMDk0YTJjNTg1OGEudHh0&__;!!F9svGWnIaVPGSwU!- > fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG > st67j-3O5xU$ > > > > > -----Original Message----- > From: john.e.gr...@wellsfargo.com.INVALID > <john.e.gr...@wellsfargo.com.INVALID> > Sent: 03 February 2022 19:33 > To: users@tomcat.apache.org > Subject: RE: help with high cpu usage > > Alan, > > > > -----Original Message----- > > From: Alan F <shiva...@hotmail.com> > > Sent: Thursday, February 03, 2022 12:19 PM > > To: Tomcat Users List <users@tomcat.apache.org> > > Subject: help with high cpu usage > > > > Had some issues today with one prod host. One is fine the other has > > stuck around 80%Cpu. > > Ive taken a thread dump from both hosts and would appreciate someone > > give a once over what it may be before I kill and restart. They are > > clustered nodes so running identical apps and loadbalanced by a hardware > balancer. > > > > Node1 is ok (relatively!) > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > > > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS1hODllYzBkZS01OGJjLTQ > > > 2ZDQtYWRhNS1kYjkxZjM2NjI1ZTAudHh0&__;!!F9svGWnIaVPGSwU!91fvMc > > RzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-mDKbTl_dmUSa0LaAolXlhipl- > > Fk2pQ$ > > > > > > Node 2 still has high CPU usage > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > > > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yN2I0YWY4Mi05OWFhL > > > TQ3YjYtOGQ2My0wMDMwZjlkNDQzNjMudHh0&__;!!F9svGWnIaVPGSwU!9 > > 1fvMcRzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn- > > mDKbTl_dmUSa0LaAolXlhipkmLFr3E$ > > Taking thread dumps was a good idea but I see very little activity on node 2. > It looks like two threads are in Hibernate's > EntityEntryContext.downgradeLocks() method, one is taking the thread > dump itself, and another is reading a request from a client, I think. I would > not expect this to add up to 80% CPU usage unless one of those threads is > stuck in a loop. Comparing multiple thread dumps taken 5-10 seconds apart > would help answer this question. How much GC is there? Could these > Hibernate queries be pulling a huge amount of data from the DB, thus > causing a lot of GC activity? > > Node 1 looks idle except for the thread taking the thread dump. > > Do you know for sure that it's the Tomcat process that is using the CPU? Of the 3 dumps from the same node, the first two are 3 hours apart, yet the two query threads seem to still be doing the same thing. (I verified that the timestamps in the dumps are different but maybe I made a mistake.) They are both making the Hibernate downgradeLocks call. This is occurring in the context of committing a transaction. That definitely makes it look like those threads are somehow either stuck or are looping. The 3rd dump has those same two threads actually making queries to Oracle. So within about 10 seconds, we have two threads committing transactions followed by making additional queries. Definitely show these to the developers if you haven't already. I didn't follow what you said about the GC. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
