I don't think a cpu loop will make tomcat stopping responding to requests I will make it very slow to respond But a shortage on memory is hard to recover
Jeff Le vendredi 28 septembre 2012, mailingl...@j-b-s.de a écrit : > Maybe an infinite loop? We observed something similar due to a bug in the > java regex impl and certain user input causes this regex looping behaviour. > As this locked one core but to the user it simply looked like server was > not responding, guess what happend? Right: they press refresh page and next > core gone :-) > So running state might be evil, too. > Switch on GC logging, maybe its just related to a full gc. In case this > happens again take a thread dump and search similarities. Take multiple > dumps in a 5 sec interval. Try to find long running threads (in our case we > noticed the regex bug) > > Jens > > Sent from my iPhone > > On 27.09.2012, at 22:05, Kirill Kireyev <kir...@instagrok.com> wrote: > > > Thanks for all the advice everyone! There is a possibility that the CPU > is caused by an app thread - I am looking into that possibility. Will let > you know when I find out more. > > > > Thanks, > > Kirill > > > > On 9/27/12 12:17 PM, Shanti Suresh wrote: > >> Hi Kirill, > >> > >> Like Mark, Bill and Jeff said, those threads are normal > request-processing > >> threads. I have included a script that might help with isolating high > CPU > >> issues with Tomcat. > >> > >> Also, I think it might be helpful to see how the Java heap is > performing as > >> well. > >> Please bring up Jconsole and let it run over the week. Inspect the > graphs > >> for Memory, CPU and threads. Since you say that high CPU occurs > >> intermittently several times during the week and clears itself, I > wonder if > >> it is somehow related with the garbage collection options you are using > for > >> the server. Or it may be a code-related problem. > >> > >> Things to look at may include: > >> > >> (1) Are high CPU times related to Java heap reductions happening at the > >> same time? ==> GC possibly needs tuning > >> (2) Are high CPU times related to increase in thread usage? ==> > possible > >> livelock in looping code? > >> (3) how many network connections come into the Tomcat server during > >> high-CPU times? Possible overload-related? > >> > >> Here is the script. I made a couple of small changes, for e.g., > changing > >> the username. But didn't test it after the change. During high-CPU > times, > >> invoke the script a few times, say 30 seconds apart. And then compare > the > >> thread-dumps. I like to use TDA for thread-dump analysis of Tomcat > >> thread-dumps. > >> > >> Mark, et al, please feel free to help me refine this script. I would > like > >> to have a script to catch STUCK threads too :-) Let me know if anyone > has > >> a script already. Thanks. > >> > >> --------------high_cpu_diagnostics.pl:----- > >> #!/usr/bin/perl > >> # > >> > >> use Cwd; > >> > >> # Make a dated directory for capturing current diagnostics > >> my ($sec,$min,$hour,$mday,$mon,$year, > >> $wday,$yday,$isdst) = localtime time; > >> $year += 1900; > >> $mon += 1; > >> my $pwd = cwd(); > >> my $preview_diag_dir = > "/tmp/Preview_Diag.$year-$mon-$mday-$hour:$min:$sec"; > >> print "$preview_diag_dir\n"; > >> mkdir $preview_diag_dir, 0755; > >> chdir($preview_diag_dir) or die "Can't chdir into $preview_diag_dir > $!\n"; > >> > >> # Capture Preview thread dump > >> my $process_pattern = "preview"; > >> my $preview_pid = `/usr/bin/pgrep -f $process_pattern`; > >> my $login = getpwuid($<) ; > >> if (kill 0, $preview_pid){ > >> #Possible to send a signal to the Preview Tomcat - either > "webinf" > >> or "root" > >> my $count = kill 3, $preview_pid; > >> }else { > >> # Not possible to send a signal to the VCM - use "sudo" > >> system ("/usr/bin/sudo /bin/kill -3 $preview_pid"); > >> } > >> > >> # Capture Preview thread dump > >> system ("/usr/bin/jmap > >> -dump:format=b,file=$preview_diag_dir/preview_heapdump.hprof > $preview_pid"); > >> > >> # Gather the top threads; keep around for reference on what other > threads > >> are running > >> @top_cmd = ("/usr/bin/top", "-H", "-n1", "-b"); > >> @sort_cmd = ("/bin/sort", "-r", "-n", "-k", "9,9"); > >> @sed_cmd = ("/bin/sed", "-n", "'8,$p'"); > >> system("@top_cmd 1> top_all_threads.log"); > >> > >> # Get your tomcat user's threads, i.e. threads of user, "webinf" > >> system('/usr/bin/tail -n+6 top_all_threads.log | /bin/sort -r -n -k > "9,9" | > >> /bin/grep webinf top_all_threads.log 1> top_user_webinf_threads.log'); > >> > >> # Get the thread dump> <instaGrok_sml.png> > -- Jeff MAURY "Legacy code" often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury