You might try running the jvm under gdb. I've had mixed success with that in the past, but it might be worth a try. Then when it hangs you might be able to interrupt gdb and poke around at the threads.
JD -----Original Message----- From: Poppe, Troy [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 14, 2004 12:56 PM To: Jboss User (E-mail) Subject: Re: [JBoss-user] JBoss 3.2.3 problems running in Linux for z/Seri es Adrian, Thanks for the reply. Couple of comments to your response. 1. Unfortunately, when the instance of Linux crashes, it crashes hard. No response to any terminal input, no new terminals, nothing. So diagnosing what's going on is turning into a scenario where we gather some info, and then make it crash again. (You've heard the joke about the hardware engineer, mechanical engineer, and a software engineer in a car with no brakes going down a hill?) That output that I attached from 'top' was all that was left just moments before the crash. The fact that all the values for RSS and SIZE being the same seems to be standard operating procedure for how top reports child process (like the ones that Java spawn). 2. The 'crash' was at the Linux instance level, equivalent to your x86 box crashing. Unfortunately, we've only seen one small message in the kernel log, and that was from kswapd. No dump files have been generated. It appears from IBM's diagnostic guide for the 1.4.1 SDK that there is an environment variable "JAVA_DUMP_OPTS" that I may need to set. The question is, is the JVM crashing first, or is the linux instance crashing first? 3. This Linux instance I have is a test instance to see what the requirements are for an application server in a z/800. The IBM recommended approach for these instances is to starve the instance of physical memory that cannot be shared amongst instances, and then force the Linux kernel to swap into, what is effectively, shared swap memory. The z/800 then swaps the shared swap memory to disk as it feels it needs to. All in all, the design of the virtual machining in this beast is a bit over my head, and I bow to the VM gods to have it work. It is entirely possible that this is a memory starvation issue. (The question for me is then, why does JBoss run for hours with no problem, and suddenly everything crashes? What changed?) 4. Is there any way that you know of to get JBoss to convince the JVM that it needs to do a system-wide gc? Is it as simple as calling System.gc()? Thanks for your input! Troy ------------------ Subject: Re: [JBoss-user] JBoss 3.2.3 problems running in Linux for z/Series From: Adrian Brock <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Organization: Date: 14 Jan 2004 19:38:09 +0000 Reply-To: [EMAIL PROTECTED] First have you tried kill -3 886 to get a threaddump? 886 is the pid of one the java threads. This will show you stacktraces of the current active threads on the console. You said it crashed, did the output from the crash include a threaddump? Second, it is generally a bad idea to run a java virtual machine with lots of swap. Periodically the VM does a full garbage collection which requires it to look at the whole heap (and it probably compacts it afterwards). The performance of this will be very bad if it has to swap memory to and from disk. Third, you are looking at threads. ps and top don't understand the linux threading model. Each java thread is a linux process that shares the same memory. Try using pstree -p to get a better idea of what is going on. The overall java process has 16844 SIZE and 11M RSS. The clue is that they are all the same numbers. :-) Regards, Adrian On Wed, 2004-01-14 at 19:04, Poppe, Troy wrote: > I am running into a curious problem running JBoss on SLES8 31-bit on VM4.3. The > instance has been allocated 128Mb of physical memory and approx 512Mb of swap. > We have the VM configured to use QDIO with the OSA adapter in the z/800. > > I'm currently testing this problem using IBMJava2-s390-131 and IBMJava2-s390-141. > > The problem I am experiencing is that the instance the JBoss container is running > on will, after JBoss has been up and running (and unused) for sometime, hit 100% > CPU usage, and there is no way to regain access or control of the box short of a > hard-restart. > > We believe we have narrowed the problem down to the java virtual machine and/or > JBoss 3.2.3. The following is output from top that was left running before the > instance crashed. You'll note that the JBoss java processes have run amok. > > [ -- snip -- ] > > 12:29am up 10:38, 1 user, load average: 20.43, 20.28, 19.85 > 89 processes: 67 sleeping, 22 running, 0 zombie, 0 stopped > CPU states: 5.1% user, 94.8% system, 0.0% nice, 0.0% idle > Mem: 126064K av, 123448K used, 2616K free, 0K shrd, 8148K buff > Swap: 575584K av, 77184K used, 498400K free 12544K cached > > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND > 886 jboss 25 0 16844 11M 140 R 5.7 9.6 5:55 java > 887 jboss 25 0 16844 11M 140 R 5.7 9.6 5:53 java > 893 jboss 25 0 16844 11M 140 R 5.7 9.6 6:03 java > 913 jboss 25 0 16844 11M 140 R 5.7 9.6 5:50 java > 918 jboss 25 0 16844 11M 140 R 5.7 9.6 5:50 java > 920 jboss 25 0 16844 11M 140 R 5.7 9.6 5:50 java > 925 jboss 25 0 16844 11M 140 R 5.7 9.6 5:49 java > 928 jboss 25 0 16844 11M 140 R 5.7 9.6 5:53 java > 929 jboss 25 0 16844 11M 140 R 5.7 9.6 5:56 java > 932 jboss 25 0 16844 11M 140 R 5.7 9.6 5:53 java > 933 jboss 25 0 16844 11M 140 R 5.7 9.6 5:54 java > 934 jboss 25 0 16844 11M 140 R 5.7 9.6 5:54 java > 941 jboss 25 0 16844 11M 140 R 5.7 9.6 5:53 java > 955 jboss 25 0 16844 11M 140 R 5.3 9.6 5:51 java > 944 jboss 25 0 16844 11M 140 R 4.2 9.6 5:52 java > 2612 root 16 0 880 840 652 R 3.6 0.6 20:04 top > 892 jboss 25 0 16844 11M 140 R 2.8 9.6 5:51 java > > [ -- snip -- ] > > At this point, we are persuing the path of trying to determine what, if anything, > JBoss is doing outside of a user request. We are trying to determine 1) why are > the processes for JBoss in the running state, it should be entirely idle; 2) why > does the SIZE of the java process differ from when we first start JBoss (roughly > 73600); 3) why does the RSS differ from when we first start JBoss (roughly 71M). > > Basically, I'm curious if anyone in the JBoss community is running JBoss with the > IBM JVM on Linux for z/Series successfully. > > Any help is greatly appreciated. > > Troy Poppe > > ------------------------------------------------------- This SF.net email is sponsored by: Perforce Software. Perforce is the Fast Software Configuration Management System offering advanced branching capabilities and atomic changes on 50+ platforms. Free Eval! http://www.perforce.com/perforce/loadprog.html _______________________________________________ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user ------------------------------------------------------- This SF.net email is sponsored by: Perforce Software. Perforce is the Fast Software Configuration Management System offering advanced branching capabilities and atomic changes on 50+ platforms. Free Eval! http://www.perforce.com/perforce/loadprog.html _______________________________________________ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
