Please post a jstack of the pid 10 seconds apart. That will reveal all. On Thu, 08 Aug 2019, 20:22 Coty Sutherland, <csuth...@apache.org> wrote:
> I'd suggest writing a small script to loop about 10 times and capture top > and thread dumps with jstack at the same time, then wait a few seconds then > repeat. After that you can grab the pid/tid from the top output and compare > that with your thread dump to see exactly what the thread is doing for the > iteration/duration you specify. > > Other questions that I haven't seen asked, how long does the CPU usage > persist? Is it only at startup or does it randomly start after some uptime? > Have your webapps or dependencies changed around the time the issue > started? Do the working and nonworking servers run the same webapps with > the same workload? > > On Thu, Aug 8, 2019 at 2:09 PM Eric Robinson <eric.robin...@psmnv.com> > wrote: > > > Utkarsh and John, thank you for your feedback. > > > > Since everything was originally on Windows, and we built a new Linux > > server with fresh tomcat installs, and the only thing we moved over from > > the old Windows servers was the tomcat application folder itself, and the > > 100% CPU problem persisted, I can't imagine what else could be causing it > > except the tomcats, but I'm open to ideas. > > > > When it happens, all the tomcats start using high CPU at the same time. > > See the following top output. > > > > top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67, > > 34.89 > > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie > > %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, > > 0.0 st > > KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 > buff/cache > > KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail > Mem > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java > > 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java > > 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java > > 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java > > 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java > > 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java > > 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java > > 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java > > 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java > > 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java > > 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java > > 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java > > 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java > > 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java > > 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15 > > ksoftirqd/7 > > 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop > > 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31 > > kworker/7:0 > > > > I'll check the localhost_access logs and see if something suspicious > > stands out. > > > > --Eric > > > > > > -----Original Message----- > > From: Utkarsh Dave <utkarshkd...@gmail.com> > > Sent: Thursday, August 8, 2019 12:33 PM > > To: Tomcat Users List <users@tomcat.apache.org> > > Subject: Re: Tomcat Server Using 100% CPU > > > > Did you reviewed the localhost_access log file. Which web-application is > > using tomcat the most ? > > > > On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <eric.robin...@psmnv.com> > > wrote: > > > > > We have a farm of VMs, each running multiple instances of tomcat (up > > > to 80 instances per server). Everything has been running fine for > > > years, but recently one server has started nailing the CPU to 100% > > utilization. > > > > > > We have tried: > > > > > > > > > * Different versions of tomcat and JDK > > > * Doubling the resources to 16 cores and 56 GB RAM > > > * Moving the VM to different physical server > > > * Rebuilding the tomcat instances on a brand new VM using Windows > > > Server 2019 > > > * Rebuilding the tomcat instances on a brand new VM using Red Hat > > > Enterprise Linux 7.5 > > > > > > Nothing has worked. No matter where we run the tomcats, they drive CPU > > > up to 100%. Meanwhile the other six servers are still running fine. > > > They all run the same canned tomcat applications. > > > > > > We would appreciate some guidance on getting to the bottom of this > > problem. > > > > > > --Eric > > > > > > > > > Disclaimer : This email and any files transmitted with it are > > > confidential and intended solely for intended recipients. If you are > > > not the named addressee you should not disseminate, distribute, copy or > > alter this email. > > > Any views or opinions presented in this email are solely those of the > > > author and might not represent those of Physician Select Management. > > > Warning: Although Physician Select Management has taken reasonable > > > precautions to ensure no viruses are present in this email, the > > > company cannot accept responsibility for any loss or damage arising > > > from the use of this email or attachments. > > > > > Disclaimer : This email and any files transmitted with it are > confidential > > and intended solely for intended recipients. If you are not the named > > addressee you should not disseminate, distribute, copy or alter this > email. > > Any views or opinions presented in this email are solely those of the > > author and might not represent those of Physician Select Management. > > Warning: Although Physician Select Management has taken reasonable > > precautions to ensure no viruses are present in this email, the company > > cannot accept responsibility for any loss or damage arising from the use > of > > this email or attachments. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > >
