Re: [JBoss-user] More Out of Memory Errors!
Were are battling this even as I type... It seem that if jBoss hits the operating systems pid (thread) limit for a process, it throws out of memory exceptions. We just ran some load test scripts and when we get too many threads, pop goes the weasil. I'm looking at two possibilities... Catch the out of memory exceptions in my code and then sleep for a bit and try again (not a great olution) or throw more hardware / operating system at it. Maybe both. Does anyone know what jBoss should do when the thread limit is hit? Thanks... Jim --On Thursday, June 21, 2001 10:11 AM +0100 Nicolai P Guba [EMAIL PROTECTED] wrote: Hello The problem is much more drastic than I initially thought. I just asked a question about closing PreparedStatements. Obviously, I had a few errors. I left the server running, recompiled the app, redeployed, forgot to remove one close on the stmt, then redeployed and now this is what I've got. [Auto deploy] Deployment failed:file:/usr/local/jboss-2.2.2/deploy/CAMS.jar [Auto deploy] org.jboss.deployment.J2eeDeploymentException: Error while starting CAMS.jar: unable to create new native thread [Auto deploy]at org.jboss.deployment.J2eeDeployer.startApplication(J2eeDeployer.java:448) [Auto deploy] at org.jboss.deployment.J2eeDeployer.deploy(J2eeDeployer.java:178) [Auto deploy] at java.lang.reflect.Method.invoke(Native Method) [Auto deploy] at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628) [Auto deploy] at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523) [Auto deploy] at org.jboss.ejb.AutoDeployer.deploy(AutoDeployer.java:358) [Auto deploy] at org.jboss.ejb.AutoDeployer.run(AutoDeployer.java:221) [Auto deploy] at java.lang.Thread.run(Thread.java:484) *ouch* It appears that it doens't only occur when it doesn't validate/verify, but also when there are transaction rollback exceptions. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user I shall be telling this with a sigh Somewhere ages and ages hence: Two roads diverged in a wood, and I - I took the one less traveled by, And that has made all the difference. - Robert Frost, 1916 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] More Out of Memory Errors!
Maybe I can throw a little light on this. I don't know too much about
what goes on inside jBoss or the JVM, but I can tell you what I see
happening on a Linux system.
When something is wrong with the deployable EAR, and I am not really sure
exactly what, jBoss can get into a state where it just starts creating
threads until the operating system puts a stop to it. In Linux, each
thread gets a process identifier ('pid') and threads are 'lightweight
processes' which run on the operating system scheduler. The threads share
memory and descriptors so the overhead is much less than for heavyweight
processes, which is the whole purpose of threads.
However, the system limitations applicable to heavyweight processes apply
equally to lightweight processes, in particular that a single non-root
user may never exceed half of the available system pids. By default when
building a Linux 2.2 kernel, the system has a global limit of 512 pids.
This is configurable in the linux source (include/linux/tasks.h):
#define NR_TASKS512 /* On x86 Max about 4000 */
#define MAX_TASKS_PER_USER (NR_TASKS/2)
#define MIN_TASKS_LEFT_FOR_ROOT 4
It is possible on most platforms which run Linux to increase this number
in the source code and rebuild the kernel. (It is problematic on SPARC,
but that it beyond the scope of the discussion here.) Nevertheless, it is
important to realize that (a) there is always some finite limit which
usually has a grounding in actual hardware, (b) scheduling lots of threads
requires time and effort on the part of the kernel which is by definition
done at the expense of useful work, and (c) Linux typically runs on large
servers using only the defaults such that increasing the task count is so
rare that it requires a source edit rather than a configuration option.
As long as jBoss is not run by the root user, which is obviously extremely
dangerous and undesirable for other reasons anyway, the maximum number of
pids it can own concurrently is going to be limited to 255 on a default
kernel. Once this thread limit is reached, jBoss will essentially wedge
to the point where it no longer even responds to SIGTERM to shut down
gracefully, but requires SIGKILL to stop it. I think that jBoss sees this
situation as an out of memory condition as reflected by the JVM, but I
am not sure about this. Of course, if jBoss is run by the root user for
some reason, the entire machine will be wedged at this point and the only
remedy would be something as drastic as the reset switch or the watchdog.
These observations apply to JBoss-2.2.1/Tomcat-3.2.1 running on Debian
(Woody) using Java HotSpot(TM) Server VM (build 1.3.1-b24, mixed mode).
The Linux kernel is 2.2.19 SMP. The run.sh file has been modified to
include the switches -Xms8m -Xmx480m so as to allow the JVM to allocate
much more than the default 64MB, but in fact it usually only allocates
between 45MB and 65MB before wedging. The machine has 256MB physical RAM
and 1GB of swap, and in practice never swaps. Here is a snapshot with
jBoss running under moderate load:
total used free sharedbuffers cached
Mem:257640 191672 65968 73588 77708 22528
-/+ buffers/cache: 91436 166204
Swap: 1048560 01048560
The main thread of the JVM at this point shows:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
jboss19954 0.6 25.3 923052 65184 ? S05:01 0:36
Notice that the operating system virtual machine size of 923MB is a little
large, but this is just address space and is fairly far divorced from
actual memory. The real resident size of the JVM is only 65MB, and the
whole machine has only about 200MB allocated and none of the 1GB swap
space in use at all. I am not sure why the JVM has such a large amount of
address space allocated, even if not actual memory, when it has been told
that it can use only about half of that. Remember that the data above
comes from a snapshot of a normally running jBoss installation. However,
maybe out of memory means exactly what it says.
-- Mike
On 2001-06-21 at 05:43 -0400, Jim Archer wrote:
Were are battling this even as I type...
It seem that if jBoss hits the operating systems pid (thread) limit for a
process, it throws out of memory exceptions. We just ran some load test
scripts and when we get too many threads, pop goes the weasil.
I'm looking at two possibilities... Catch the out of memory exceptions in
my code and then sleep for a bit and try again (not a great olution) or
throw more hardware / operating system at it.
Maybe both.
Does anyone know what jBoss should do when the thread limit is hit?
Thanks...
Jim
--On Thursday, June 21, 2001 10:11 AM +0100 Nicolai P Guba
[EMAIL PROTECTED] wrote:
Hello
The problem is much more drastic than I initially thought. I just
asked a question about closing PreparedStatements. Obviously, I had a
Re: [JBoss-user] More Out of Memory Errors!
MB == Michael Bilow [EMAIL PROTECTED] writes: MB However, the system limitations applicable to heavyweight MB processes apply equally to lightweight processes, in particular MB that a single non-root user may never exceed half of the available MB system pids. By default when building a Linux 2.2 kernel, the MB system has a global limit of 512 pids. This is configurable in MB the linux source (include/linux/tasks.h): MB #define NR_TASKS 512 /* On x86 Max about 4000 */ #define MB MAX_TASKS_PER_USER (NR_TASKS/2) #define MIN_TASKS_LEFT_FOR_ROOT 4 A, h, Thank you for reminding me of that valued filename! grin MB As long as jBoss is not run by the root user, which is obviously MB extremely dangerous and undesirable for other reasons anyway, the MB maximum number of pids it can own concurrently is going to be MB limited to 255 on a default kernel. Once this thread limit is MB reached, jBoss will essentially wedge to the point where it no MB longer even responds to SIGTERM to shut down gracefully, but MB requires SIGKILL to stop it. I think that jBoss sees this MB situation as an out of memory condition as reflected by the JVM, MB but I am not sure about this. Of course, if jBoss is run by the MB root user for some reason, the entire machine will be wedged at MB this point and the only remedy would be something as drastic as MB the reset switch or the watchdog. I agree 101% with your assessment. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] More Out of Memory Errors!
JA == Jim Archer [EMAIL PROTECTED] writes: JA Were are battling this even as I type... It seem that if jBoss JA hits the operating systems pid (thread) limit for a process, it JA throws out of memory exceptions. We just ran some load test JA scripts and when we get too many threads, pop goes the weasil. Yeah. I've encountered this problem when I build a Unified Messaging platform on a GNU/Linux architecture. The voice switch needed a number of threads per timeslot on the primary rate ISDN cards. This means a magnitude of 30xthread_num x 4 (ports) x num_of_cards. Not sure what the default process size was. It's either 256 or 512. One can change it by editing a header file and recompiling the kernel (forgot which file though. bummer). I wonder why jboss needs so many threads though. Can one tweak this at startup? JA I'm looking at two possibilities... Catch the out of memory JA exceptions in my code and then sleep for a bit and try again (not JA a great olution) or throw more hardware / operating system at it. JA Maybe both. JA Does anyone know what jBoss should do when the thread limit is JA hit? Hmmm, I don't know why it needs so many threads. There is very little activity on the box (dedicated JBoss devel box). Maybe one should look at the thread management issue. I've noticed that on deploy error and redeploy the number of threads increases drastically (measured this very primitively via watch 'ps ax | wc -l'). Hope this helps. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
