RE: Urgent - Issue with Tomcat/mod_jk?

[Venkat Reddy Valluri]

Hi,
   Here I am  getting "kernel out of memery killing java process" in  syslog 
messages(/var/log/messages)   not in catalina.out.  Here  I tried with max heap size 
too(-Xms256m -Xmx256m), ending with no result.
 What I thoght is sun jvm is not properly handling memory allocation/dealloctaion 
on redhat 7.2/7.3, so  I tried with IBM jvm 1.4 on redhat 7.2 with tomcat 4.0.4. 
Surprisingly I am not getting broken pipe errors as well as tomcat crashing problems

   Thank you very much for your response
Thks,
 --Venkat
 


-Original Message-
From:   Ben Ricker [mailto:[EMAIL PROTECTED]]
Sent:   Thu 12/26/2002 9:42 AM
To: Tomcat Users List
Cc: 
Subject:    RE: Urgent - Issue with Tomcat/mod_jk?
I beleive we are dealing with a different problem. My app has been
running on Tomcat 4.0.6 for weeks until I had the problem recently.
There was no crash.

What happened was, it seems, a user REALLY fat-fingered the enter key or
some smart person sent the same request to the server 180 times within
one minute. I traced back through my access logs for Apache and caught
the massive number if requests for one URL at the same time I had the
problem.

All of these requests overwhelmed the number of AJP13 processors I had
configured (well, they are still at the default setting) and also caused
a massive opening of database connections.

Anyway, as to your problem: You may not have enough memory setup for
your JVM. Do you send amx -Xm and -Xs parameters to Tomcat using
JAVA_OPTS? You may be getting more load then Tomcat can handle due to a
RAM constraint.

Ben Ricker


On Tue, 2002-12-24 at 13:39, Venkat Reddy Valluri wrote:
> Hi ,
>Even I too getting the same problem(lots of broken pipe errors )with catalina 
>4.0.4(redhat 7.3, j2sdk1.4.1) connecting  apache 2.0.40(redat 7.3) on diffrenet 
>machine with mod_jk,
>  But what happened is after it ran for cuople of hours, tomcat crases,
>  giving kernel out of memory error in syslog messages 
> 
> Did you get any workaround for this
> Thks,
> --Venkat
>   
> 
> 
> -Original Message-
> From: Ben Ricker [mailto:[EMAIL PROTECTED]]
> Sent: Mon 12/23/2002 5:49 PM
> To:   Tomcat Users List
> Cc:   
> Subject:  Re: Urgent - Issue with Tomcat/mod_jk?
> I had suspected that in the problem earlier in the day (the part of the
> email I posted at the end if the last email you replied to). However,
> what did not make sense was the connection pool woth the database also
> going up.
> 
> If I understand the stuff below, after 85 concurrent connections (max
> connections + accept count) I would start getting the "Out of
> Processors" error. However, I cannot see how this would cause the
> DATABASE connection pool to grow as large as it did (we usually handle
> 50k connections a business day with 5 pooled connections; during the
> earlier problem, the database connections went to *30*, our maximum).
> 
> Additionally, why would we contine to get the following error:
> 
> 2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
> java.io.IOException: Broken pipe
> at java.net.SocketOutputStream.socketWrite(Native Method)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
> at org.apache.ajp.Ajp13.send(Ajp13.java:525)
> at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
> at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
> at 
>org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
> at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
> at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
> at java.lang.Thread.run(Thread.java:479)
>  
> without seeing the "Out of Processor" msgs? 
> 
> I may be looking at two independent problems: one a scalability issue
> with the ajp13 processors (which are at the ddefault setting, btw) and
> some other issue I am in the dark about.
> 
> Anyway, these questions are rhetorical. Thanks for the reply and do not
> feel obliged to answer back. I need to start putting pressure on the
> developers to help me out here.
> 
> Thanks again,
> 
> Ben Ricker
> 
> On Mon, 2002-12-23 at 16:07, Mark Eggers wrote:
> > Ben,
> > 
> > Disclaimer:
> > 
> > I'm not a Tomcat developer, but I do use it to develop
> > software and integrate applications.
> > 
> > In $CATALINA_HOME/conf/server.xml you should see an
> > entry similar to the following:
> > 
> > 
> >  >  className="org.apache.ajp.tomcat4.Ajp13Connector"
> >  port="8009"
> >  minProcessors="5"
> >  maxProcessors="75"
> >  acceptCount="

RE: Urgent - Issue with Tomcat/mod_jk?

[Ben Ricker]

I beleive we are dealing with a different problem. My app has been
running on Tomcat 4.0.6 for weeks until I had the problem recently.
There was no crash.

What happened was, it seems, a user REALLY fat-fingered the enter key or
some smart person sent the same request to the server 180 times within
one minute. I traced back through my access logs for Apache and caught
the massive number if requests for one URL at the same time I had the
problem.

All of these requests overwhelmed the number of AJP13 processors I had
configured (well, they are still at the default setting) and also caused
a massive opening of database connections.

Anyway, as to your problem: You may not have enough memory setup for
your JVM. Do you send amx -Xm and -Xs parameters to Tomcat using
JAVA_OPTS? You may be getting more load then Tomcat can handle due to a
RAM constraint.

Ben Ricker


On Tue, 2002-12-24 at 13:39, Venkat Reddy Valluri wrote:
> Hi ,
>Even I too getting the same problem(lots of broken pipe errors )with catalina 
>4.0.4(redhat 7.3, j2sdk1.4.1) connecting  apache 2.0.40(redat 7.3) on diffrenet 
>machine with mod_jk,
>  But what happened is after it ran for cuople of hours, tomcat crases,
>  giving kernel out of memory error in syslog messages 
> 
> Did you get any workaround for this
> Thks,
> --Venkat
>   
> 
> 
> -Original Message-
> From: Ben Ricker [mailto:[EMAIL PROTECTED]]
> Sent: Mon 12/23/2002 5:49 PM
> To:   Tomcat Users List
> Cc:   
> Subject:  Re: Urgent - Issue with Tomcat/mod_jk?
> I had suspected that in the problem earlier in the day (the part of the
> email I posted at the end if the last email you replied to). However,
> what did not make sense was the connection pool woth the database also
> going up.
> 
> If I understand the stuff below, after 85 concurrent connections (max
> connections + accept count) I would start getting the "Out of
> Processors" error. However, I cannot see how this would cause the
> DATABASE connection pool to grow as large as it did (we usually handle
> 50k connections a business day with 5 pooled connections; during the
> earlier problem, the database connections went to *30*, our maximum).
> 
> Additionally, why would we contine to get the following error:
> 
> 2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
> java.io.IOException: Broken pipe
> at java.net.SocketOutputStream.socketWrite(Native Method)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
> at org.apache.ajp.Ajp13.send(Ajp13.java:525)
> at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
> at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
> at 
>org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
> at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
> at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
> at java.lang.Thread.run(Thread.java:479)
>  
> without seeing the "Out of Processor" msgs? 
> 
> I may be looking at two independent problems: one a scalability issue
> with the ajp13 processors (which are at the ddefault setting, btw) and
> some other issue I am in the dark about.
> 
> Anyway, these questions are rhetorical. Thanks for the reply and do not
> feel obliged to answer back. I need to start putting pressure on the
> developers to help me out here.
> 
> Thanks again,
> 
> Ben Ricker
> 
> On Mon, 2002-12-23 at 16:07, Mark Eggers wrote:
> > Ben,
> > 
> > Disclaimer:
> > 
> > I'm not a Tomcat developer, but I do use it to develop
> > software and integrate applications.
> > 
> > In $CATALINA_HOME/conf/server.xml you should see an
> > entry similar to the following:
> > 
> > 
> >  >  className="org.apache.ajp.tomcat4.Ajp13Connector"
> >  port="8009"
> >  minProcessors="5"
> >  maxProcessors="75"
> >  acceptCount="10" debug="0"/>
> > 
> > I'm using 4.1.18, so your entry may be a bit
> > different.  It looks like from your error messages
> > that you may be running out of processors.
> > 
> > The following information is taken from Tomcat's
> > 4.1.18 documentation.  If you have the documentation
> > installed somewhere, the URL is:
> > 
> > http://localhost/tomcat-docs/config/jk.html
> > 
> > acceptCount:
> > 
> > The maximum queue length for incoming connection
> > requests when all possible request processing threads
> > are in use. Any requests received when the queue is
> > full will be refused. The default val

RE: Urgent - Issue with Tomcat/mod_jk?

[Venkat Reddy Valluri]

Hi ,
   Even I too getting the same problem(lots of broken pipe errors )with catalina 
4.0.4(redhat 7.3, j2sdk1.4.1) connecting  apache 2.0.40(redat 7.3) on diffrenet 
machine with mod_jk,
 But what happened is after it ran for cuople of hours, tomcat crases,
 giving kernel out of memory error in syslog messages 

Did you get any workaround for this
Thks,
--Venkat
  


-Original Message-
From:   Ben Ricker [mailto:[EMAIL PROTECTED]]
Sent:   Mon 12/23/2002 5:49 PM
To: Tomcat Users List
Cc: 
Subject:Re: Urgent - Issue with Tomcat/mod_jk?
I had suspected that in the problem earlier in the day (the part of the
email I posted at the end if the last email you replied to). However,
what did not make sense was the connection pool woth the database also
going up.

If I understand the stuff below, after 85 concurrent connections (max
connections + accept count) I would start getting the "Out of
Processors" error. However, I cannot see how this would cause the
DATABASE connection pool to grow as large as it did (we usually handle
50k connections a business day with 5 pooled connections; during the
earlier problem, the database connections went to *30*, our maximum).

Additionally, why would we contine to get the following error:

2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
 
without seeing the "Out of Processor" msgs? 

I may be looking at two independent problems: one a scalability issue
with the ajp13 processors (which are at the ddefault setting, btw) and
some other issue I am in the dark about.

Anyway, these questions are rhetorical. Thanks for the reply and do not
feel obliged to answer back. I need to start putting pressure on the
developers to help me out here.

Thanks again,

Ben Ricker

On Mon, 2002-12-23 at 16:07, Mark Eggers wrote:
> Ben,
> 
> Disclaimer:
> 
> I'm not a Tomcat developer, but I do use it to develop
> software and integrate applications.
> 
> In $CATALINA_HOME/conf/server.xml you should see an
> entry similar to the following:
> 
> 
>   className="org.apache.ajp.tomcat4.Ajp13Connector"
>  port="8009"
>  minProcessors="5"
>  maxProcessors="75"
>  acceptCount="10" debug="0"/>
> 
> I'm using 4.1.18, so your entry may be a bit
> different.  It looks like from your error messages
> that you may be running out of processors.
> 
> The following information is taken from Tomcat's
> 4.1.18 documentation.  If you have the documentation
> installed somewhere, the URL is:
> 
> http://localhost/tomcat-docs/config/jk.html
> 
> acceptCount:
> 
> The maximum queue length for incoming connection
> requests when all possible request processing threads
> are in use. Any requests received when the queue is
> full will be refused. The default value is 10.
> 
> maxProcessors:
> 
> The maximum number of request processing threads to be
> created by this Connector, which therefore determines
> the maximum number of simultaneous requests that can
> be handled. If not specified, this attribute is set to
> 20.
> 
> NOTE:For Apache 1.3 on Unix there is a 1 to 1 mapping
> between httpd processes and Ajp13Processors. You must
> configure maxProcessors to be greater than or equal to
> the maximum number of httpd processes your Apache web
> server spawns.
> 
> minProcessors:
> 
> The number of request processing threads that will be
> created when this Connector is first started. This
> attribute should be set to a value smaller than that
> set for maxProcessors. The default value is 5.
> 
> I hope this gets you started on a productive path.
> 
> /mde/
> 
> just my two cents . . . .
> 
> __
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
> 
> --
> To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
-- 
Ben Ricker <[EMAIL PROTECTED]>
Wellinx.com


--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>





<>--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

Re: Urgent - Issue with Tomcat/mod_jk?

[Ben Ricker]

I had suspected that in the problem earlier in the day (the part of the
email I posted at the end if the last email you replied to). However,
what did not make sense was the connection pool woth the database also
going up.

If I understand the stuff below, after 85 concurrent connections (max
connections + accept count) I would start getting the "Out of
Processors" error. However, I cannot see how this would cause the
DATABASE connection pool to grow as large as it did (we usually handle
50k connections a business day with 5 pooled connections; during the
earlier problem, the database connections went to *30*, our maximum).

Additionally, why would we contine to get the following error:

2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
 
without seeing the "Out of Processor" msgs? 

I may be looking at two independent problems: one a scalability issue
with the ajp13 processors (which are at the ddefault setting, btw) and
some other issue I am in the dark about.

Anyway, these questions are rhetorical. Thanks for the reply and do not
feel obliged to answer back. I need to start putting pressure on the
developers to help me out here.

Thanks again,

Ben Ricker

On Mon, 2002-12-23 at 16:07, Mark Eggers wrote:
> Ben,
> 
> Disclaimer:
> 
> I'm not a Tomcat developer, but I do use it to develop
> software and integrate applications.
> 
> In $CATALINA_HOME/conf/server.xml you should see an
> entry similar to the following:
> 
> 
>   className="org.apache.ajp.tomcat4.Ajp13Connector"
>  port="8009"
>  minProcessors="5"
>  maxProcessors="75"
>  acceptCount="10" debug="0"/>
> 
> I'm using 4.1.18, so your entry may be a bit
> different.  It looks like from your error messages
> that you may be running out of processors.
> 
> The following information is taken from Tomcat's
> 4.1.18 documentation.  If you have the documentation
> installed somewhere, the URL is:
> 
> http://localhost/tomcat-docs/config/jk.html
> 
> acceptCount:
> 
> The maximum queue length for incoming connection
> requests when all possible request processing threads
> are in use. Any requests received when the queue is
> full will be refused. The default value is 10.
> 
> maxProcessors:
> 
> The maximum number of request processing threads to be
> created by this Connector, which therefore determines
> the maximum number of simultaneous requests that can
> be handled. If not specified, this attribute is set to
> 20.
> 
> NOTE:For Apache 1.3 on Unix there is a 1 to 1 mapping
> between httpd processes and Ajp13Processors. You must
> configure maxProcessors to be greater than or equal to
> the maximum number of httpd processes your Apache web
> server spawns.
> 
> minProcessors:
> 
> The number of request processing threads that will be
> created when this Connector is first started. This
> attribute should be set to a value smaller than that
> set for maxProcessors. The default value is 5.
> 
> I hope this gets you started on a productive path.
> 
> /mde/
> 
> just my two cents . . . .
> 
> __
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
> 
> --
> To unsubscribe, e-mail:   
> For additional commands, e-mail: 
-- 
Ben Ricker <[EMAIL PROTECTED]>
Wellinx.com


--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Re: Urgent - Issue with Tomcat/mod_jk?

[Mark Eggers]

Ben,

Disclaimer:

I'm not a Tomcat developer, but I do use it to develop
software and integrate applications.

In $CATALINA_HOME/conf/server.xml you should see an
entry similar to the following:




I'm using 4.1.18, so your entry may be a bit
different.  It looks like from your error messages
that you may be running out of processors.

The following information is taken from Tomcat's
4.1.18 documentation.  If you have the documentation
installed somewhere, the URL is:

http://localhost/tomcat-docs/config/jk.html

acceptCount:

The maximum queue length for incoming connection
requests when all possible request processing threads
are in use. Any requests received when the queue is
full will be refused. The default value is 10.

maxProcessors:

The maximum number of request processing threads to be
created by this Connector, which therefore determines
the maximum number of simultaneous requests that can
be handled. If not specified, this attribute is set to
20.

NOTE:For Apache 1.3 on Unix there is a 1 to 1 mapping
between httpd processes and Ajp13Processors. You must
configure maxProcessors to be greater than or equal to
the maximum number of httpd processes your Apache web
server spawns.

minProcessors:

The number of request processing threads that will be
created when this Connector is first started. This
attribute should be set to a value smaller than that
set for maxProcessors. The default value is 5.

I hope this gets you started on a productive path.

/mde/

just my two cents . . . .

__
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

--
To unsubscribe, e-mail:   
For additional commands, e-mail: 

Urgent - Issue with Tomcat/mod_jk?

[Ben Ricker]

I posted a message earlier which has not received any replies so I will
try another tact. The earlier mail is posted below this one.

I have a production issue that has arisen in a setup of two Apache
1.3.27/Redhat 7.3 servers load-balanced across 4 Tomcat 4.0.6/Sun
Solaris servers. Tomcat is running with Sun's 1.3 JVM.

We have gotten a lot of the following errors in the "engine" log for
Tomcat:

2002-12-23 15:21:56 Ajp13Processor[12009][16] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)

We have also gotten calls from customers saying they are getting hanging
screens in our application. We see no anomolies on out database,
although earlier we did.

Can anyone tell me what the error above is saying? Is it not able to
write to the mod_jk? It looks like it is failing in some sort of closing
routine (org.apache.ajp.tomcat4.Ajp13Response.finishResponse). Could
this be a protocol issue?

I am getting heat from my bosses on this one and I need to say
SOMETHING! Please help if you can give me any clues or suggest
possibilities.

Thanks,

Ben Ricker


-Earlier Post-
We saw a strange production issue this morning that seems to be related
to Apache/mod_jk but I wanted to see if anyone can shed some lght on it.

First, the environment:

2 Redhat 7.3 Linux server talking to 2 Sun Solaris 8 Tomcat 4.0.6
servers. Each server has 2 separate Tomcat instances for a total of 4.
The Java uses Oracles JDBC drivers (not through Tomcat, but through a
custom connection pool) to talk to an Oracle DB runnin on AIX 4.x.

The first symptom was a huge spike in DB usage due to the creation of
around 35 pooled connections to the database, This led us to think
something happened on the database side, but it seems that the pool only
grew on one of the Tomcat instances; the other instances showed the
default pool size. The Oracle database did not show any locked tables or
huge, hanging queries. 

The only problems I see in logs are related to mod_jk and Tomcat.

First in the Tomcat engine log,I see:

2002-12-23 09:07:14 Ajp13Connector[12009] No processor available, rejecting this 
connection

There are hundreds of these messages. Then I start seeing this message
in the engine log:

2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)

About a minute after, in the mod_jk logs, I see:

[Mon Dec 23 09:08:29 2002]  [jk_ajp_common.c (679)]: ajp_connection_tcp_get_message: 
Error - jk_tcp_socket_recvfull failed
[Mon Dec 23 09:08:29 2002]  [jk_ajp_common.c (1041)]: Error reading reply
[Mon Dec 23 09:08:29 2002]  [jk_ajp_common.c (1178)]: In jk_endpoint_t::service, 
ajp_get_reply failed in send loop 0

This goes on for about 20 seconds before I see a huge number of the
following:

[Mon Dec 23 09:08:49 2002]  [jk_ajp_common.c (961)]: Error ajp_process_callback - 
write failed

I am at a loss at what might cause this. Could it be related to
soemthing in Tomcat? What exactly does this sequence of events tell me?

Any light one can shed would be greatly appreciated.

Ben Ricker

-- 
Ben Ricker <[EMAIL PROTECTED]>
Wellinx.com


--
To unsubscribe, e-mail:   
For additional commands, e-mail: