HI John, Thankyou kindly for taking the time here.
First of all looking back at the historical chart for the day I observed node 2 rise in cpu activity about 11am this morning and stayed high node1 has dropped back to idle. ##Do you know for sure that its the Tomcat process that is using the CPU?## A simple top -i showing the only process still running here at 189% is a java process under tomcat account. 24667 tomcat 20 0 365.6g 17.2g 23080 S 189.7 54.9 2663:45 java I have taken 2 dumps here 10 secs apart if this helps at all? https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0zNzY2ZmZmNy0wZDgyLTRhZTItYmE3Mi0zMWQyYTYwN2M1ZjgudHh0&; https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yMzFjOGYzZS01MjYzLTRiYWYtOTJjMS1jN2RiMWRlZmEwN2QudHh0&; We do have Prometheus hooked in via jmx with Grafana but im struggling to see anything glaring apart from GC I see old Gen taking about 5hrs before it drops back one node2 as opposed to 3 on node 1. -----Original Message----- From: john.e.gr...@wellsfargo.com.INVALID <john.e.gr...@wellsfargo.com.INVALID> Sent: 03 February 2022 19:33 To: users@tomcat.apache.org Subject: RE: help with high cpu usage Alan, > -----Original Message----- > From: Alan F <shiva...@hotmail.com> > Sent: Thursday, February 03, 2022 12:19 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: help with high cpu usage > > Had some issues today with one prod host. One is fine the other has > stuck around 80%Cpu. > Ive taken a thread dump from both hosts and would appreciate someone > give a once over what it may be before I kill and restart. They are > clustered nodes so running identical apps and loadbalanced by a hardware > balancer. > > Node1 is ok (relatively!) > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS1hODllYzBkZS01OGJjLTQ > 2ZDQtYWRhNS1kYjkxZjM2NjI1ZTAudHh0&__;!!F9svGWnIaVPGSwU!91fvMc > RzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-mDKbTl_dmUSa0LaAolXlhipl- > Fk2pQ$ > > > Node 2 still has high CPU usage > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yN2I0YWY4Mi05OWFhL > TQ3YjYtOGQ2My0wMDMwZjlkNDQzNjMudHh0&__;!!F9svGWnIaVPGSwU!9 > 1fvMcRzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn- > mDKbTl_dmUSa0LaAolXlhipkmLFr3E$ Taking thread dumps was a good idea but I see very little activity on node 2. It looks like two threads are in Hibernate's EntityEntryContext.downgradeLocks() method, one is taking the thread dump itself, and another is reading a request from a client, I think. I would not expect this to add up to 80% CPU usage unless one of those threads is stuck in a loop. Comparing multiple thread dumps taken 5-10 seconds apart would help answer this question. How much GC is there? Could these Hibernate queries be pulling a huge amount of data from the DB, thus causing a lot of GC activity? Node 1 looks idle except for the thread taking the thread dump. Do you know for sure that it's the Tomcat process that is using the CPU?
