Platform: Intel PIII 797.499MHz, 256MB RAM
Debian Woody GNU/Linux,
kernel 2.2.19
Blackdown J2SDK 1.3.1
apache 1.3.19
tomcat 3.2.3
When using httperf[1] to stresstest JSPs on the above
platform, I'm running into the OutOfMemoryError
exception problem:
<http://www.jguru.com/faq/view.jsp?EID=64889>
The solution suggested in the URL above, is to increase
the heap size, but this is just pushing the problem
ahead. The important thing for us, is that tomcat doesn't die.
If this is a problem for high load sites, JSP technology
would be unusable, and obviously it isn't. So what
do others do?
Possible alternatives:
1. increase the heap size to something astronomical,
and just hope for the best?
2. tune apache/tomcat to restart more often, so that
the max heap size aren't reached before the tomcat
processes are restarted?
3. catch the exception?
4. use some kind of watchdog to restart tomcat?
Question 3 raises more questions:
3a. What do you do when you catch the exception?
There isn't much you _can_ do if you don't have
any memory left. Maybe force a garbage collection
and then re-run the request?
3b. Why do all the tomcat processes connected to
apache processes die, if one of them gets an
OutOfMemoryError exception
Which brings us to question 2: tuning apache/tomcat:
When reading the documentation, I thought that there
was supposed to be a single tomcat process, serving
all requests. However top reports a lot of tomcat
processes, when I'm stresstesting the server, which
leads me to belive that there is a separate tomcat
process for each apache server process running.
Is this correct?
In any case; based on this assumption, I reduced
the number of currently running apache processes
from the default 150 to 20, with dramatically
changed behaviour: instead of running with a 20%
load, the machine was running at an 80%-90% load
with less physical memory used (restarting processes
is a lot of work).
However, even with the high load, the end result
was a lot more stable. Watching the tomcat processes
in top, I could watch them grow up to around 82MB,
before there suddenly was a lot of processes with
around 19MB.
It wasn't completely stable though. I set up
httperf to do 1 million requests, before I left
last night, and it only got to 51927 before tomcat
died.
I run httperf from a different machine (PIII 800MHz),
using the command:
httperf --server=no-video2 --port=80 --uri=/tps-sba/Welcome.jsp
--rate 150 --num-conns=1000000 --num-calls=1
(--num-conns varies. I have mostly been using
10000).
The memory usage observed in top, also raises some
questions related to my question 1:
1a. Why does tomcat processes always start at around
19MB, no matter what I've set the initial heap
size to be?
1b. Why does the initial heap size seem to matter
for my stability if the processes always start
at 19MB? Is it all in my head?
I'm interested in all direct experiences, ideas,
guesses, answers, etc.
Thanx!
- Steinar
[1] <http://freshmeat.net/projects/httperf>
This email, its content and any attachments is PRIVATE AND CONFIDENTIAL to
TANDBERG Television. If received in error please notify the sender and
destroy the original message and attachments.
