Maybe an infinite loop? We observed something similar due to a bug in the java
regex impl and certain user input causes this regex looping behaviour. As this
locked one core but to the user it simply looked like server was not
responding, guess what happend? Right: they press refresh page and next core
gone :-)
So running state might be evil, too.
Switch on GC logging, maybe its just related to a full gc. In case this happens
again take a thread dump and search similarities. Take multiple dumps in a 5
sec interval. Try to find long running threads (in our case we noticed the
regex bug)
Jens
Sent from my iPhone
On 27.09.2012, at 22:05, Kirill Kireyev <kir...@instagrok.com> wrote:
> Thanks for all the advice everyone! There is a possibility that the CPU is
> caused by an app thread - I am looking into that possibility. Will let you
> know when I find out more.
>
> Thanks,
> Kirill
>
> On 9/27/12 12:17 PM, Shanti Suresh wrote:
>> Hi Kirill,
>>
>> Like Mark, Bill and Jeff said, those threads are normal request-processing
>> threads. I have included a script that might help with isolating high CPU
>> issues with Tomcat.
>>
>> Also, I think it might be helpful to see how the Java heap is performing as
>> well.
>> Please bring up Jconsole and let it run over the week. Inspect the graphs
>> for Memory, CPU and threads. Since you say that high CPU occurs
>> intermittently several times during the week and clears itself, I wonder if
>> it is somehow related with the garbage collection options you are using for
>> the server. Or it may be a code-related problem.
>>
>> Things to look at may include:
>>
>> (1) Are high CPU times related to Java heap reductions happening at the
>> same time? ==> GC possibly needs tuning
>> (2) Are high CPU times related to increase in thread usage? ==> possible
>> livelock in looping code?
>> (3) how many network connections come into the Tomcat server during
>> high-CPU times? Possible overload-related?
>>
>> Here is the script. I made a couple of small changes, for e.g., changing
>> the username. But didn't test it after the change. During high-CPU times,
>> invoke the script a few times, say 30 seconds apart. And then compare the
>> thread-dumps. I like to use TDA for thread-dump analysis of Tomcat
>> thread-dumps.
>>
>> Mark, et al, please feel free to help me refine this script. I would like
>> to have a script to catch STUCK threads too :-) Let me know if anyone has
>> a script already. Thanks.
>>
>> --------------high_cpu_diagnostics.pl:-----
>> #!/usr/bin/perl
>> #
>>
>> use Cwd;
>>
>> # Make a dated directory for capturing current diagnostics
>> my ($sec,$min,$hour,$mday,$mon,$year,
>> $wday,$yday,$isdst) = localtime time;
>> $year += 1900;
>> $mon += 1;
>> my $pwd = cwd();
>> my $preview_diag_dir = "/tmp/Preview_Diag.$year-$mon-$mday-$hour:$min:$sec";
>> print "$preview_diag_dir\n";
>> mkdir $preview_diag_dir, 0755;
>> chdir($preview_diag_dir) or die "Can't chdir into $preview_diag_dir $!\n";
>>
>> # Capture Preview thread dump
>> my $process_pattern = "preview";
>> my $preview_pid = `/usr/bin/pgrep -f $process_pattern`;
>> my $login = getpwuid($<) ;
>> if (kill 0, $preview_pid){
>> #Possible to send a signal to the Preview Tomcat - either "webinf"
>> or "root"
>> my $count = kill 3, $preview_pid;
>> }else {
>> # Not possible to send a signal to the VCM - use "sudo"
>> system ("/usr/bin/sudo /bin/kill -3 $preview_pid");
>> }
>>
>> # Capture Preview thread dump
>> system ("/usr/bin/jmap
>> -dump:format=b,file=$preview_diag_dir/preview_heapdump.hprof $preview_pid");
>>
>> # Gather the top threads; keep around for reference on what other threads
>> are running
>> @top_cmd = ("/usr/bin/top", "-H", "-n1", "-b");
>> @sort_cmd = ("/bin/sort", "-r", "-n", "-k", "9,9");
>> @sed_cmd = ("/bin/sed", "-n", "'8,$p'");
>> system("@top_cmd 1> top_all_threads.log");
>>
>> # Get your tomcat user's threads, i.e. threads of user, "webinf"
>> system('/usr/bin/tail -n+6 top_all_threads.log | /bin/sort -r -n -k "9,9" |
>> /bin/grep webinf top_all_threads.log 1> top_user_webinf_threads.log');
>>
>> # Get the thread dump
>> my @output=`/usr/bin/jstack -l ${preview_pid}`;
>> open (my $file, '>', 'preview_threaddump.txt') or die "Could not open file:
>> $!";
>> print $file @output;
>> close $file;
>>
>> open LOG, "top_user_webinf_threads.log" or die $!;
>> open (STDOUT, "| tee -ai top_cpu_preview_threads.log");
>> print "PID\tCPU\tMem\tJStack Info\n";
>> while ($l = <LOG>) {
>> chop $l;
>> $pid = $l;
>> $pid =~ s/webinf.*//g;
>> $pid =~ s/ *//g;
>> ## Hex PID is available in the Sun HotSpot Stack Trace */
>> $hex_pid = sprintf("%#x", $pid);
>> @values = split(/\s+/, $l);
>> $pct = $values[8];
>> $mem = $values[9];
>> # Debugger breakpoint:
>> $DB::single = 1;
>>
>> # Find the Java thread that corresponds to the thread-id from the TOP output
>> for my $j (@output) {
>> chop $j;
>> ($j =~ m/nid=$hex_pid/) && print $hex_pid . "\t" . $pct . "\t" .
>> $mem . "\t" . $j . "\n";
>> }
>> }
>>
>> close (STDOUT);
>>
>> close LOG;
>>
>>
>> ------------------end of script --------------------------
>>
>> Thanks.
>>
>> -Shanti
>>
>>
>> On Thu, Sep 27, 2012 at 2:11 PM, Bill Miller <
>> millebi.subscripti...@gmail.com> wrote:
>>
>>> I agree; we have reproducible instances where PermGen is not set to our
>>> requirements on the Tomcat startup parameters and it will cause a "lockup"
>>> every time. Do some JMX monitoring and you may discover a memory spike
>>> that's killing Tomcat.
>>>
>>> Bill
>>> -----Original Message-----
>>> From: Jeff MAURY [mailto:jeffma...@gmail.com]
>>> Sent: September-27-2012 2:01 PM
>>> To: Tomcat Users List
>>> Subject: Re: high CPU usage on tomcat 7
>>>
>>> This is probably due to out of memory, I have the same problem on my ubuntu
>>> ci machine Did you monitor your tomcat with jmx ?
>>>
>>> Jeff
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>
>
>
> --
> Kirill Kireyev, PhD
> Founder/CTO instaGrok.com
> kir...@instagrok.com
> Twitter: @instaGrok
> FB: facebook.com/instagrok
> <instaGrok_sml.png>
