Re: [EXTERNAL] RE: Re: General Architecture Question for multiple websites on a single RedHat server
We hit the leap second when it happened and rebooted the servers which solved that problem. Thanks anyway On Mon, Jul 9, 2012 at 5:24 PM, Martin Gainty mgai...@hotmail.com wrote: apparently for RH 6.1 stopping and starting the ntp daemon works /etc/init.d/ntpd stop; date; date `date +%m%d%H%M%C%y.%S`; date ; /etc/init.d/ntpd starthttps://bugzilla.redhat.com/show_bug.cgi?id=836748 Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. Date: Mon, 9 Jul 2012 22:42:21 +0200 From: a...@ice-sa.com To: users@tomcat.apache.org Subject: Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server Simon, Leonard wrote: Well our Tomcat went out to lunch again and we had to recycle the webserver to get things stablized. By this I mean we get reports from the users that screens become unresponsive and looking at a top we see tomcat process taking 100% CPU. Was able to do a thread dump captured with a kill -3 PID and here it is if anyone is so inclined to comment on it. Have not followed this thread and not looked at the thread dump (which would not tell me anything anyway), but by any chance have you googled for leap second bug ? Mentioning that because of your 100% CPU above. See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=479765 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
Chris, Thanks for looking at this. Tomcat version is 6.0.32. mod_jk is at 1.2.31 Someone else did the thread dump so I'm assuming they did it on the right process. On Tue, Jul 10, 2012 at 12:19 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Simon, On 7/9/12 4:24 PM, Simon, Leonard wrote: Well our Tomcat went out to lunch again and we had to recycle the webserver to get things stablized. By this I mean we get reports from the users that screens become unresponsive and looking at a top we see tomcat process taking 100% CPU. Are you sure this is the right process? Was able to do a thread dump captured with a kill -3 PID and here it is if anyone is so inclined to comment on it. This thread dump shows a mostly-idle server with the exception of those threads in socketAccept() (not sure why these count as RUNNABLE when they are really blocking) and those executing reads from the client connection(s). What exact version of Tomcat are you using, and what version of mod_jk (or, if you are using mox_proxy_ajp, what httpd version)? IIRC, there have been some stability improvements in recent Tomcat versions around the worker threads being returned to their associated connectors. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/8VfwACgkQ9CaO5/Lv0PCgWwCfbN/E29q4DKee4q1A+IEMmED6 8+0AnivucFDMS/7lhPiOb+0tv5I/6vim =ABXf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
- Original Message - From: Simon, Leonard leonard.si...@hsn.net To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Tuesday, July 10, 2012 9:54 AM Subject: Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server Chris, Thanks for looking at this. Tomcat version is 6.0.32. mod_jk is at 1.2.31 Someone else did the thread dump so I'm assuming they did it on the right process. On Tue, Jul 10, 2012 at 12:19 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Simon, On 7/9/12 4:24 PM, Simon, Leonard wrote: Well our Tomcat went out to lunch again and we had to recycle the webserver to get things stablized. By this I mean we get reports from the users that screens become unresponsive and looking at a top we see tomcat process taking 100% CPU. Are you sure this is the right process? Was able to do a thread dump captured with a kill -3 PID and here it is if anyone is so inclined to comment on it. This thread dump shows a mostly-idle server with the exception of those threads in socketAccept() (not sure why these count as RUNNABLE when they are really blocking) and those executing reads from the client connection(s). What exact version of Tomcat are you using, and what version of mod_jk (or, if you are using mox_proxy_ajp, what httpd version)? IIRC, there have been some stability improvements in recent Tomcat versions around the worker threads being returned to their associated connectors. - -chris I didn't see much in the way that rang immediate alarm bells. It looks like you're processing about 18 client connections, and everything else is pretty quiet. These client connections are going through the AJP connector (as you've noted in your reply above). A few things though: As someone in this thread has already mentioned, permgen is pretty full. You might try increasing that with -XX:MaxPermSize=128m. There are a lot of garbage collection threads. You can see this on a multi-core system. From digging around, it appears that the number of parallel garbage collection threads follows this formula: 8 + (5/8)X = GCT You get one GCT (garbage collection thread) per core for the first 8 cores, and then 5/8 of a thread for every core after that. So in your case: 8 + (5/8)X = 18 X = 16 This means that your system has 24 cores. Are you running on a 24 core system, or have you tuned garbage collection with JVM arguments. In general, if you're not running into GC issues, tuning GC parameters is counter-productive. If you do have to tune GC parameters, lots of testing is in order. I noticed that you also have an MQ Trace monitor running. Are you using MQ? Directly accessing an MQ service without going through a pool configured for graceful restarts / retries can cause a system to become unresponsive. However, I don't see any evidence of that in this thread dump. As I've said off line, it's really difficult to see what's consuming CPU from one thread dump. Here's how to start figuring out what is going on with your system. 1. Keep access logs If you don't, then start. You'll want the access logs to replay on a test environment to see if you can recreate the problem. JMeter is a good tool for replaying information from access logs. 2. When the problem occurs a. Multiple thread dumps, about 5 seconds apart. Use a tool like jstack so it's scriptable jstack -l [process-id] where [process-id] is the process id of the distressed Tomcat The -l generates a long listing, and may not be necessary. You'll need to have the right permissions (either root or the user running the JVM being targeted with the process id). b. At the same time use something like the following to see which thread is consuming CPU: ps -L -o pcpu,lwp -p [process-id] where [process-id] is the process id of the distressed Tomcat This will show all the threads of the process, the percentage of CPU used for each thread, and the thread process ID. You can then correlate the thread process ID with the thread dump to see exactly what is consuming the CPU. This will generate tons of output, so it's best to put both in a script and direct the output to files. Now you'll end up with the following: 1. What requests were being made of your server when the problem occurred 2. Multiple thread dumps while the problem is occurring 3. The identity of the thread (or threads) that is consuming the CPU Once you get this information, you'll be in a much better position to determine what is causing your problems. . . . . just my two cents. /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
Mark Eggers wrote: ... . . . . just my two cents. As a consultant in my professional capacity, I'd charge just about 100,000 times more for that. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
- Original Message - From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Tuesday, July 10, 2012 12:12 PM Subject: Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server Mark Eggers wrote: ... . . . . just my two cents. As a consultant in my professional capacity, I'd charge just about 100,000 times more for that. Writing quick how-to mail messages on a mailing list - cheap. As a consultant in my professional capacity, writing the scripts, running the analysis, replaying the access log on a test system, helping / dealing with developers, system admins, system architects, management . . . agreed. Oh, and document, document, document. (but please to always call it research - apologies to Tom Lehrer) /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
