[JBoss-user] [Performance Tuning] - Re: JBoss effectively hangs due to massive context switching
"lafr" wrote : What are the memory settings for Xmx ? | When the JVM reaches this limit, garbage collections occur all the time and consume all available cpu time. | /usr/local/jdk/bin/java -server -Xms1024m -Xmx1536m -Djava.awt.headless=true -Dprogram.name=run.sh -XX:PermSize=256M -XX:MaxPermSize=256 This process runs as the jboss user, so if the CPU is spiking due to the garbage collector, why does vmstat report the CPU spike under the "sy" column and not the "us" column? Doesn't the garbage collector run inside the jboss-owned JVM? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=316#316 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=316 --- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - JBoss effectively hangs due to massive context switching
We have had a recurring problem over the past few months where every once in a while (1-5 times a month), our production app server (running JBoss) will become extremely slow, with vmstat reporting several hundred thousand context switches per second and maxed out cpus (dual processor machine) split about 25/75 between user/system processes. When it first starts happening, JBoss still seems to respond to most requests (albeit very slowly), but others just timeout, and eventually it reaches a point where nearly all requests timeout. At that point, it is impossible to do much more than to log in to the server and kill -9 the jboss process, and even that takes about 5 minutes to accomplish. As soon as the jboss process is killed, the entire system goes back to normal, and we can restart jboss and live happily again until it happens again after about a week of uptime. Here is what things look like right before restarting jboss: procs memory swap io system cpu r b swpd free buff cache si sobibo incs us sy id wa 12 0 100952 16412 282520 124440000 0 0 106 306339 17 83 0 0 13 0 100952 16408 282520 124440000 024 117 297674 23 77 0 0 15 0 100952 16408 282520 124440000 0 0 108 336135 17 83 0 0 16 0 100952 16408 282520 124440000 0 0 108 159630 20 80 0 0 15 0 100952 16408 282520 124440000 0 0 116 176452 24 76 0 0 14 0 100952 16408 282520 124440000 0 0 116 99453 27 73 0 0 15 0 100952 16416 282520 124440000 024 117 96588 27 73 0 0 And here is what it looks like right after killing jboss: 1 0 91632 1409604 282520 124443600 0 0 171 130 0 0 100 0 1 0 91632 1409604 282520 124443600 0 176 17868 0 1 99 1 1 0 91632 1409604 282520 124443600 0 0 12538 0 0 100 0 And here is what it looks like right when jboss has started back up again: 5 0 91632 1139332 286124 132144400 0 0 329 636 76 2 22 0 7 0 91632 1133860 286124 132144800 0 0 447 839 98 1 1 0 4 0 91632 1133520 286124 132146800 0 0 1284 2436 96 3 1 0 3 0 91632 1131932 286124 132146800 0 292 2388 4477 86 4 10 0 6 0 91632 1131932 286160 13214680020 0 814 1428 94 1 4 1 2 0 91632 1121720 286160 132146800 0 0 272 504 97 1 2 0 We are running JBoss 4.0.2, but this was happening on 3.2.3 also (we upgraded hoping this would go away, and it hasn't). The OS is "Red Hat Enterprise Linux ES release 3 (Taroon Update 5)", and the JVM is build 1.4.2_08-b03. One possibility that I have been suspicious of is our use of temporary files that are marked "deleteOnExit". We use them a lot, and store them all in one directory. As time goes on, their number only increases. The last time this happened, there were about 64,000 temporary files in the directory. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3888668#3888668 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3888668 --- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Tomcat, HTTPD, Servlets & JSP] - Re: JBoss4.0.x getRequestURI Return Different Result from 3.
Regarding the change in getRequestURL, I found this discussion: http://issues.apache.org/bugzilla/show_bug.cgi?id=28222 So evidently this change is here to stay. View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3888499#3888499 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3888499 --- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Tomcat, HTTPD, Servlets & JSP] - Re: JBoss4.0.x getRequestURI Return Different Result from 3.
I just noticed that getRequestURL() also returns something different between 3.2.3 and 4.0.2 when called from a jsp; e.g.: 3.2.3 getRequestURL() = /aServletName/path/info 4.0.2 getRequestURL() = /path/to/the/file.jsp Is this intentional? From a JSP, how can I get the original request URL? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3888497#3888497 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3888497 --- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - Re: idle time goes to 0 in an instant and stays that way
"genman" wrote : | I would turn on debug logging and get stack traces. If you get a feeling for what your process normally does during the time it changes behavior, a trace might pinpoint something. | | Also, check using "lsof" and see if your process is running out of files. Thanks for the suggestions. Our application does use (lots of) temporary files quite intensively, but if there was a file handle problem, shouldn't there be something about it recorded in a log file somewhere? View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3876542#3876542 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3876542 --- This SF.Net email is sponsored by: NEC IT Guy Games. Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20 ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - Re: idle time goes to 0 in an instant and stays that way
Oh, and: Linux version 2.4.21-4.0.1.ELsmp ([EMAIL PROTECTED]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #1 SMP Thu Oct 23 01:21:42 EDT 2003 View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3876475#3876475 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3876475 --- This SF.Net email is sponsored by: NEC IT Guy Games. Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20 ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - Re: idle time goes to 0 in an instant and stays that way
"[EMAIL PROTECTED]" wrote : - which JVM and which OS (and threading model)? Red Hat Enterprise Linux ES release 3 (Taroon) java version "1.4.2_03" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02) Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode) JBoss 3.2.3 View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3876473#3876473 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3876473 --- This SF.Net email is sponsored by: NEC IT Guy Games. Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20 ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - idle time goes to 0 in an instant and stays that way
First 1 processor went to 0% idle and then 73 seconds later the 2nd processor also went to 0%, and stayed down. Looking at our vmstat logs, this happened yesterday at 2:06 PM, but amazingly, nobody complained about the system being unusually slow until this afternoon, at which point we restarted JBoss. Here are the facts: - Throughout this time, the system still responded to requests, most of the time very quickly, but sometimes not at all. - vmstat shows an almost constant 0% idle, the only exceptions being about 10 blibs per minute going up to 7% idle, and going to 50% idle for 2.5 seconds at exactly-60-second intervals. - We have a two processor machine. - At the time we restarted, the JVM was running about 490 threads. - We have maxProcessors set to 200. - Along with the drop to 0% idle, the context switches reported by vmstat jumped from an average of about 1000 per second to an average of about 300,000 per second. - This is the first time this has happened on a system that has been running under JBoss for 2.5 years, with steadily increasing traffic. - Last application patch was a minor one two days ago; last major patch was 2 months ago. - DB load and web server load during this whole time was normal. What could it be? An infinite loop in newly deployed code? A JBoss configuration setting that's reached a threshhold? An OS setting? Many thanks, nwc View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3876362#3876362 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3876362 --- This SF.Net email is sponsored by: NEC IT Guy Games. Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20 ___ JBoss-user mailing list JBoss-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Performance Tuning] - Re: Intermittent crashes
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3821311#3821311 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3821311 "mikestephen" wrote : Hi, The crashes occur seemingly randomly, it might stay up for 6 weeks before crashing, or sometimes only 2 hours. I am now having this exact same problem with JBoss 3.2.3. Did you ever resolve this problem? What was the resolution? Thanks, Nathan --- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click ___ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
[JBoss-user] [Connectors and JCA/JBoss] - JBoss 3.2.3 crashing and hanging unpredictably
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3821306#3821306 Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3821306 We recently began experiencing crashing and hanging of JBoss 3.2.3 in our production environment at unpredictable time intervals--anywhere from 30 minutes to 12 hours after re-running the app. This began after we upgraded our mysql connector from 3.0.6 to 3.0.10, and made several minor application changes. We have sinced rolled back all of those changes, but continue to experience the same crashing and hanging. Sometimes it hangs with one CPU maxed out, sometimes it hangs with both CPUs idle (we have a two CPU machine). Sometimes it kills itself with no core dump and no trace in the logs, and sometimes it kills itself by shutting itself down and printing this to server.log: 2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,756 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,756 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,765 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,766 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate 2004-02-12 09:16:29,766 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate Our configuration is like this: JBoss 3.2.3 java version "1.4.2_03" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02) Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode) Linux version 2.4.21-4.0.1.ELsmp ([EMAIL PROTECTED]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #1 SMP Thu Oct 23 01:21:42 EDT 2003 mysql-connector-java-3.0.10-stable-bin.jar mysql Ver 11.18 Distrib 3.23.58, for redhat-linux-gnu (i386) This configuration had worked fine for us (except using the 3.0.6 mysql connector) for several weeks without incident. --- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click ___ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
