I don't think a cpu loop will make tomcat stopping responding to requests
I will make it very slow to respond
But a shortage on memory is hard to recover
Jeff
Le vendredi 28 septembre 2012, mailingl...@j-b-s.de a écrit :
> Maybe an infinite loop? We observed something similar due to a bug in the
> java regex impl and certain user input causes this regex looping behaviour.
> As this locked one core but to the user it simply looked like server was
> not responding, guess what happend? Right: they press refresh page and next
> core gone :-)
> So running state might be evil, too.
> Switch on GC logging, maybe its just related to a full gc. In case this
> happens again take a thread dump and search similarities. Take multiple
> dumps in a 5 sec interval. Try to find long running threads (in our case we
> noticed the regex bug)
>
> Jens
>
> Sent from my iPhone
>
> On 27.09.2012, at 22:05, Kirill Kireyev <kir...@instagrok.com> wrote:
>
> > Thanks for all the advice everyone! There is a possibility that the CPU
> is caused by an app thread - I am looking into that possibility. Will let
> you know when I find out more.
> >
> > Thanks,
> > Kirill
> >
> > On 9/27/12 12:17 PM, Shanti Suresh wrote:
> >> Hi Kirill,
> >>
> >> Like Mark, Bill and Jeff said, those threads are normal
> request-processing
> >> threads. I have included a script that might help with isolating high
> CPU
> >> issues with Tomcat.
> >>
> >> Also, I think it might be helpful to see how the Java heap is
> performing as
> >> well.
> >> Please bring up Jconsole and let it run over the week. Inspect the
> graphs
> >> for Memory, CPU and threads. Since you say that high CPU occurs
> >> intermittently several times during the week and clears itself, I
> wonder if
> >> it is somehow related with the garbage collection options you are using
> for
> >> the server. Or it may be a code-related problem.
> >>
> >> Things to look at may include:
> >>
> >> (1) Are high CPU times related to Java heap reductions happening at the
> >> same time? ==> GC possibly needs tuning
> >> (2) Are high CPU times related to increase in thread usage? ==>
> possible
> >> livelock in looping code?
> >> (3) how many network connections come into the Tomcat server during
> >> high-CPU times? Possible overload-related?
> >>
> >> Here is the script. I made a couple of small changes, for e.g.,
> changing
> >> the username. But didn't test it after the change. During high-CPU
> times,
> >> invoke the script a few times, say 30 seconds apart. And then compare
> the
> >> thread-dumps. I like to use TDA for thread-dump analysis of Tomcat
> >> thread-dumps.
> >>
> >> Mark, et al, please feel free to help me refine this script. I would
> like
> >> to have a script to catch STUCK threads too :-) Let me know if anyone
> has
> >> a script already. Thanks.
> >>
> >> --------------high_cpu_diagnostics.pl:-----
> >> #!/usr/bin/perl
> >> #
> >>
> >> use Cwd;
> >>
> >> # Make a dated directory for capturing current diagnostics
> >> my ($sec,$min,$hour,$mday,$mon,$year,
> >> $wday,$yday,$isdst) = localtime time;
> >> $year += 1900;
> >> $mon += 1;
> >> my $pwd = cwd();
> >> my $preview_diag_dir =
> "/tmp/Preview_Diag.$year-$mon-$mday-$hour:$min:$sec";
> >> print "$preview_diag_dir\n";
> >> mkdir $preview_diag_dir, 0755;
> >> chdir($preview_diag_dir) or die "Can't chdir into $preview_diag_dir
> $!\n";
> >>
> >> # Capture Preview thread dump
> >> my $process_pattern = "preview";
> >> my $preview_pid = `/usr/bin/pgrep -f $process_pattern`;
> >> my $login = getpwuid($<) ;
> >> if (kill 0, $preview_pid){
> >> #Possible to send a signal to the Preview Tomcat - either
> "webinf"
> >> or "root"
> >> my $count = kill 3, $preview_pid;
> >> }else {
> >> # Not possible to send a signal to the VCM - use "sudo"
> >> system ("/usr/bin/sudo /bin/kill -3 $preview_pid");
> >> }
> >>
> >> # Capture Preview thread dump
> >> system ("/usr/bin/jmap
> >> -dump:format=b,file=$preview_diag_dir/preview_heapdump.hprof
> $preview_pid");
> >>
> >> # Gather the top threads; keep around for reference on what other
> threads
> >> are running
> >> @top_cmd = ("/usr/bin/top", "-H", "-n1", "-b");
> >> @sort_cmd = ("/bin/sort", "-r", "-n", "-k", "9,9");
> >> @sed_cmd = ("/bin/sed", "-n", "'8,$p'");
> >> system("@top_cmd 1> top_all_threads.log");
> >>
> >> # Get your tomcat user's threads, i.e. threads of user, "webinf"
> >> system('/usr/bin/tail -n+6 top_all_threads.log | /bin/sort -r -n -k
> "9,9" |
> >> /bin/grep webinf top_all_threads.log 1> top_user_webinf_threads.log');
> >>
> >> # Get the thread dump> <instaGrok_sml.png>
>
--
Jeff MAURY
"Legacy code" often differs from its suggested alternative by actually
working and scaling.
- Bjarne Stroustrup
http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury
