Re: Re: Tomcat crashes simultaneous more connection above 990+

[Eamonn Maher]

Hi folks,
coincidentally I am also having problems with large number of concurrent 
connections to Tomcat

getting errors when we try and connect many clients concurrently to Tomcat. 
Does anyone have experience of this sort of Tomcat usage or should we be 
looking at a different servlet container for heavy loads.
The setup is as follows
Tomcat 3.2.2. on WinNT SP 6 running as a standalone servlet container.
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)

We have a test environment where a number (2-4) of other machines, with 
multi-threaded clients (30-1000 threads) bombard the servlet in an effort 
to test the servlet performance. This leads to a number of problems

1) First of all the mutli-threaded test clients (also written in Java) 
occasionally throw a NoClassDefFoundError when the following line of code 
is executed

 PrintWriter out = new PrintWriter(conn.getOutputStream());

The class that is not found is a Socket class (sorry I don't have the trace 
at the moment) is not found. This happens when the number of threads on the 
client is high (c. 1000). Obviously it is not a Tomcat problem so I 
shouldn't even mention it. It is easily worked around by catching for the 
error and retrying.

It looks slightly similar to a problem in JKD 1.2 where the Calender class 
would load a ResourceBundle containing Locale info (in particular Date 
format info) and would report a MissingResourceBundle exception 
occasionally with a high number of threads creating Calenders 
simultaneously. A chain of resources are created and loaded into a 
HashTable during a call to load the resource, but some of the resources may 
be garbage collected during recursive calls to ResourceBundle.getObject()
resulting in a MissingResourceException. Since garbage collection is not 
very frequent the chances of this occurring during a resource access are 
quite low. I only mention it since it has some surface similarities with my 
problem. For more info look here
http://developer.java.sun.com/developer/bugParade/bugs/4335714.html

2) The second problem may be an actual Tomcat problem.
It is caused by the same line of code and is an IOException saying that the 
connection has been refused.  Even when we increase various parameters in 
the server.xml file -
 
 
 
 
 
to ensure that Tomcat will accept more connections it does not seem to 
work. This problem has been demonstrated to be independent of out servlet 
and appears when accessing a resource such as a JSP file. Again this is not 
a critical problem since we can catch the exception and retry the 
connection but it is puzzling as to why the high backlog figure doesn't 
ensure that the socket connection attempts are not queued. Again sorry that 
I don't have a trace to hand at the moment.

3) We have more serious problems on the Tomcat side, after some while (a 
couple of hours) with a high number (1000 -1500) of clients continuously 
connecting we get errors in the tomcat logs
"2001-08-07 09:33:47 - Ctx( /certresolver ): IOException in: R( 
/certresolver+ /certs + null) socket write error (code=10055)"

A quick search for code=10055 reveals it to be an error code thrown by 
winsock, however it covers a couple of different situations including 
trying to increase the timeout on a socket. The one that appears most 
appropriate to this problem is a bug report at sun
http://developer.java.sun.com/developer/bugParade/bugs/4474929.html

"Embedded in the exception's message string is code 10055.  Under Win32, 
this error is WSAENOBUFS which means the TCP/IP stack has temporarily run 
out of buffer space and cannot complete the write.  Microsoft has verified 
this as a bug in the WinSock TCP/IP stack (see the MSDN article ID: 
Q201213).  This would be a recoverable error if not for the way the Java 
native code handles the error.  The recovery would simply be to wait for 
the stack buffers to empty and try sending the same data again at a later 
time."

It could possibly be related to this error but without more information in 
the exception we can't tell. Has anyone ever experienced something like this?

4) The final problem is that Tomcat reports is an OutOfMemoryException 
after continuous usage with a high number of clients. Increasing the memory 
available to the Tomcat process (using -Xmx500m) does not make the problem 
go away, the taskmanager reports that the Tomcat process is only using a 
fraction of its allocated memory. Could this be a related to the previous 
discussion on upper limits for blocking I/O connections per process

Many thanks if you can help in any way.
Eamonn


>Well I think you're completely nuts :) For 10,000 concurrent requests for
>*dynamic* data (or else you'd just be using Apache.. right?), you'd need a
>server farm full of fancy equipment, a load-balancing server, and stuff that
>even my wildly ignorant speculations can't conceive.
>
>Besides, I don't thin

Re: Re: Tomcat crashes simultaneous more connection above 990+

[Jeff Turner]

Well I think you're completely nuts :) For 10,000 concurrent requests for
*dynamic* data (or else you'd just be using Apache.. right?), you'd need a
server farm full of fancy equipment, a load-balancing server, and stuff that
even my wildly ignorant speculations can't conceive.

Besides, I don't think any single server with simple thread-per-request, 
blocking IO model can handle that sort of load. A smart fellow called Matt
Welsh did his PhD thesis on highly concurrent server apps, and the results of
his thinking are here:

http://www.cs.berkeley.edu/~mdw/proj/sandstorm/

In particular, the paper "The Staged Event-Driven Architecture for Highly 
Concurrent Servers." is well worth reading.

--Jeff
(who thinks 50 concurrent users is pushing things)


On Mon, Aug 13, 2001 at 01:54:28PM +0530, Santosh Pasi wrote:
> Hi Jeff,
> 
> Thanks for reply.
> I am looking around 10,000 to 12,000 concurrent connections.
[snip]

Re: Re: Tomcat crashes simultaneous more connection above 990+

[Santosh Pasi]

Hi Jeff,

Thanks for reply.
I am looking around 10,000 to 12,000 concurrent connections.


I have few more questions
1.  i tried both mod_jk and mod_jserver, but i really didn't found any 
improvemnet.  Which is best to use.  if mod_jk then mod_jk(ajp12 or ajp13)?
What other things i can do to increase perfomance .. as it reaches 
around 1200 above connections .. its speed decrease ...
Regards,
Santosh Pasi





---Original Message--
Date: Mon, 13 Aug 2001 17:46:14 +1000
>From: Jeff Turner <[EMAIL PROTECTED]>
>To: Santosh Pasi <[EMAIL PROTECTED]>
>Cc: [EMAIL PROTECTED]
>Subject: Re: Tomcat crashes simultaneous more connection above 990+
>Mail-Followup-To: Santosh Pasi <[EMAIL PROTECTED]>, 
[EMAIL PROTECTED]>Content-Disposition: inline
>User-Agent: Mutt/1.3.19i
>
>On Mon, Aug 13, 2001 at 11:01:24AM +0530, Santosh Pasi wrote:
>> Hi,
>> 
>> Still not any reply 
>> 
>> Is there any tips for compiling tomcat for heavy load ?...
>> After changing few settings .. i can reach upto 1800 simultaneous 
>> connections ... after that tomcat ... didn't send any reply(no more 
crash)...>
>Just how high do you want it to go? I'm quite impressed it got to 990 in
>the first place :) What sort of throughput does that translate to, and
>would your service provider be able to accommodate it anyway? A segv in
>the underlying VM is not likely to be something Tomcat developers can
>fix, what with Java being closed source.
>
>--Jeff
>
>> Regards,
>> Santosh Pasi
>> 
>> 
>> 
>> ---Original Message--
>> Mailing-List: contact [EMAIL PROTECTED]; run by 
ezmlm>> >Precedence: bulk
>> >Reply-To: [EMAIL PROTECTED]
>> >List-Help: 
>> >List-Unsubscribe: 
>> >List-Post: 
>> >To: [EMAIL PROTECTED]
>> >From: Santosh Pasi<[EMAIL PROTECTED]>
>> >Cc: 
>> >Reply-To: Santosh Pasi<[EMAIL PROTECTED]>
>> >Date: Sat, 11 Aug 2001  10:14:00 +0530
>> >In-Reply-To: [EMAIL PROTECTED]
>> >Subject: Tomcat crashes simultaneous more connection above 990+
>> >
>> >Hi everyone,
>> >
>> >Can anyone know how much simultaneous connections Tomcat can 
support, >> >where do I need to change the settings to reflect same.  Is no. of 
max. >> >connections support ... is somewhat related with no. of connections 
to >> Tomcat.>Yesterday, I tried all combinations .. still as soon as total 
>> >connections .. reaches above 990 .. 997(at most) it crashes Tomcat.  
>> And no more >process of Tomcat is left.  Although Apache adn other 
applications are >> not >affected.
>> >Is there a no. limitation on child process ... as here child 
processes >> >. increases to some what 1048.. and it crashes.
>> >Same thing i tried on AIX 4.5 ... and it supported above 2600 
process >> >adn finally it crashed there.
>> >We are looking for 10,000 - 12000 simulatenous connections .. it 
means >> >1-12000 child process java ... is it problem with java .. we ar 
>> using 1.2.2.>Is there any fine tuning that i can do in Tomcat or 
Apache or Linux?>> >
>> >Regards,
>> >Santosh Pasi
>> >
>> >
>> >
>> >---Original Message--
>> >Mailing-List: contact [EMAIL PROTECTED]; run by 
ezmlm>> >>Precedence: bulk
>> >>Reply-To: [EMAIL PROTECTED]
>> >>List-Help: 
>> >>List-Unsubscribe: 
>> >>List-Post: 

>> >>To: [EMAIL PROTECTED]
>> >>From: Santosh Pasi<[EMAIL PROTECTED]>
>> >>Cc: 
>> >>Reply-To: Santosh Pasi<[EMAIL PROTECTED]>
>> >>Date: Fri, 10 Aug 2001 12:30:04 +0530
>> >>In-Reply-To: [EMAIL PROTECTED]
>> >>Subject: Re: Jakarta crashes simultaneous more connection above 990+
>> >>
>> >>Hi,
>> >>
>> >>Even changing pools setting like
>> >>Increasing the pool size resolved the problem, specifically the
>> >>following parameters (neatly copied from the tomcat users guide)
>> >>
>> >>   > >> name="max_threads"
>> >> value="200"/>
>> >> > >> name="max_spare_threads"
>> >> value="20"/>
>> >> > >> name="min_spare_threads"
>> >> value="20" />
>> >>
>> >>The values were arbitrary - I stuck mine up to max_threads = 200,
>> >>max_spare_threads=20, min_spare_threads=20
>> >>and lo - the hang problem  went away.
>> >>
>> >>Even I had used options
>> >>
>> >>TOMCAT_OPTS=" -Xms64m"
>> >>
>> >>Didn't solve the problem.
>> >>
>> >>Still same problem.
>> >>
>> >>Regards,
>> >>Santosh Pasi
>> >>
>> >>---Original Message--
>> >>Mailing-List: contact [EMAIL PROTECTED]; run by 
ezmlm>> >>>Precedence: bulk
>> >>>Reply-To: [EMAIL PROTECTED]
>> >>>List-Help: 
>> >>>List-Unsubscribe: 
>> >>>List-Post: 

>> >>>To: [EMAIL PROTECTED]
>> >>>From: Santosh Pasi<[EMAIL PROTECTED]>
>> >>>Cc: 
>> >>>Reply-To: Santosh Pasi<[EMAIL PROTECTED]>
>> >>>Date: Fri, 10