All
Apologies, this is unrelated. How do I unsubscribe from this mailing
list, I thought it would be useful and small but its overwhelming my
inbox?
Thanks in Advance.
Luke Walshe
BT Operate, HGIPCC Technical Specialist
Telephone: +44 (0)1314483482, Email: [EMAIL PROTECTED]
-----Original Message-----
From: Ahmed Musa [mailto:[EMAIL PROTECTED]
Sent: 21 February 2008 09:25
To: Tomcat Users List
Subject: Re: mod_jk Problems - - worker went to error state and dont
recover
Hello Rainer,
Thanks for your informations - the Situation gets more clear now.
I will read again some dics - following your links and will make further
tests also with the improved logging.
Thanks a lot for your time
with best regards
ahmed
-------- Original-Nachricht --------
Datum: Wed, 20 Feb 2008 18:59:01 +0100
Von: Rainer Jung <[EMAIL PROTECTED]>
An: Tomcat Users List <users@tomcat.apache.org>
Betreff: Re: mod_jk Problems - - worker went to error state and dont
recover
Ahmed Musa wrote:
Hello,
Wow -thank you very much Rainer for your very quick and informative
answer.
I will go to 1.2.26 and think about some "smoother" Values for
reply_timeout and max_reply_timeouts.
I will search for the requests which causes the Problems - becasue i
still log the response time in your mentioned way - but I am not sure
that the
Userrequests are responsible for the Situation.
One note: for Apache httpd 2.x %d is microseconds (there is no format
for milliseconds), for Tomcat %D is milliseconds. As long as you are
searching for the root cause, it might make sense to have both access
logs active to check about duration differences.
So one further question - does mod_jk itself checks if the Backend
is
reachable - without userrequests?
No. Everything only works on top of user requests.
When there are connections to the Backend - are they closed after
the
respone or are the hold open for further requests.
In general hold open. There are parameters on how long they are held
open without more requests before they get shut down, and also how
many
might be kept open even when no requests are coming in. Those are the
connection pool parameters, which you will find on
http://tomcat.apache.org/connectors-doc/reference/workers.html
Tomcat also has a connectionTimeout on the connector, which will shut
down a connection from the Tomcat side if it is idle for to long.
If you don't want to reuse connections at all, there's also a setting
(a
JkOption in Apache).
Is it possible that the Checkpoint Firewall in Between can be
responsible for the connectivity problem?
It can cut a connection that's idle for too long. Since you have
cping/cpong active via connect_timeout and prepost_timeout, you should
get a cping error message, if the connection was dropped by the
firewall
during idle times and mod_jk tries to use it again. The reply timeout
in
the error log indicates, that the backend isn't answering. Of course
if
it takes *very* long to answer, it might be that the firewall dropped
the connection in between, but then the root cause would still be the
long response time of the backend.
Another point is the "not recovering" of the worker. Yes, you are
right
- in this situation i have many reply_timeouts - but these happens in
a
period of time - for example 30 minutes - but the worker is still dead
even
then when there are no more reply_timeouts. It remains dead.
It was necessary to restart it manually via jkstatus.
I assume you are using stickyness, so when a session started on a
node,
it will stay there. So when a worker is in error for a long time, all
new sessions will start on other nodes. If the worker is ready for
recovery, it needs a request, that doesn't carry a session to get
probed
with this request.
In jkstatus, the status of an error worker should switch to REC, when
mod_jk decides that it could send a non-sticky request there (to
probe)
and to PRB, during the time this request is on the node, and finally
either to OK or back to ERR depending on the result of the request.
You can log the number of errors (and accesses) that happened on the
node in the httpd access log. If you think that the node simply stays
in
error for a long time, then the error count (and access count) should
stay constant. I would expect, that they do not.
Have a look at how LogFormat in Apache httpd works, and then add some
of
those documented in
http://tomcat.apache.org/connectors-doc/reference/apache.html
like:
JK_LB_LAST_NAME
JK_LB_LAST_ACCESSED
JK_LB_LAST_ERRORS
JK_LB_LAST_BUSY
JK_LB_LAST_STATE
using the syntax %{JK_LB_LAST_STATE}n etc.
Another point is the learning - i read the dics - the infos on the
apache Website i dont't find other ones - are there other ones ? - and
they are
not going in depth - if you read the spec and watch the logs it is -
for me
- very hard to match the things. Also the many possibilities that
mod_jk
has to prove if there is a connection to the Backend,... - i
understand them
but check the reality in an error situation is very hard. Under
matching i
mean "Which Part of the Communication sequence failed - why - and
causes
which error message".
But i will try - and study also the mailing list..
It's hard for us too (sometimes).
Thank you for your time - tomorrow we will have the new version and
will
see what happens.
best
ahmed
Regards,
Rainer
-------- Original-Nachricht --------
Datum: Wed, 20 Feb 2008 15:56:42 +0100
Von: Rainer Jung <[EMAIL PROTECTED]>
An: Tomcat Users List <users@tomcat.apache.org>
Betreff: Re: mod_jk Problems - - worker went to error state and
dont
recover
[EMAIL PROTECTED] wrote:
See Thread at: http://www.techienuggets.com/Detail?tx=25608 Posted
on
behalf of a User
Hallo to all, After long unsuccessful research i hope someone can
give me a hint to the following problems.
Our Apache-mod_jk-Tomcat Infrastructur was running without
Problems
for about one year-than since two month mod_jk errors occurs.
We upgraded the mod_jk Version, made improvements in the
worker.properties - the problems changed and get less but
sometimes
they
appear further on.
It seems that the mod_jk worker loose the connection to their
Tomcat-Backendserver - there are messages in the mod_jk log Files
which
points in this direction. Normally this seems not to be a big
problem
-
but under certain conditions (which ?) the worker goes to an error
state
and cannot recover itself- must be done manually.
Problem 1: The Tomcats are reachable - unknown why the workers
think
the
server is dead ?
Problem 2: I have no idea why the worker goes to an error state
and
cannot recover.
2 is a consequence of 1
Problem3: I miss explanations of logged messages - i read the
messages
-
but cannot match them to the situation - when does a worker post
this
messages
1 is a consequence of these messages
[Wed Feb 20 10:04:01.889 2008] [19237:3086010048] [info]
jk_handler::mod_jk.c (2270): Aborting connection for worker=ajp_ggi
[Wed Feb 20 10:04:39.799 2008] [19294:3086010048] [error]
ajp_get_reply::jk_ajp_common.c (1623): (INETP1011) Timeout with
waiting
reply from
tomcat. Tomcat is down, stopped or network problems (errno=110)
[Wed Feb 20 10:04:39.799 2008] [19294:3086010048] [error]
ajp_service::jk_ajp_common.c (2034): (INETP1011) receiving reply
from
tomcat failed with
out recovery in send loop attempt=0
[Wed Feb 20 10:04:41.799 2008] [19294:3086010048] [error]
service::jk_lb_worker.c (1105): unrecoverable error 504, request
failed. Tomcat failed in
the middle of request, we can't recover to another instance.
The second line tells us, that your configured reply_timeout fired.
You set it to 120000 (2 minutes), so there are requests taking
longer
than 2 minutes on the backend, before the first response packet
comes
back from the backend.
With your configuration mod_jk then doesn't wait any longer on the
reply
*and puts the backend into error mode*.
Up until version 1.2.25, if you use a reply-timeout, you need to
set it
to a high number which justifies the resoning "if it takes that
long,
that something is wrong with the backend".
Reality shows: there is no such number. Often there are few
requests
that take unaccetably long on the backend *although* the backend is
still working.
So in 1.2.25 we added max_reply_timeouts. With this set in addition
to
reply_timeout, mod_jk will abort waiting for a reply after
reply_timeout, but allow some timeouts before actually deciding to
put
the backend into error.
Unfortunately the implementation of max_reply_timeouts in 1.2.25
was
wrong, so you need to go to 1.2.26 to get it working right.
See:
http://issues.apache.org/bugzilla/show_bug.cgi?id=43229
Caution: this does *not* explain, why the backends are not
automatically
recovered after a minute of error condition. Maybe you have times,
where
you getr to many of those reply_timeouts (see log file), and
although
we
recover after a minute the backend almost immediately goes back
into
error status.
-> Which Timeout - how does mod_jk think Tomcat is down ? Where
can i
found details to errno=110 ?...
reply_timeout, see above and also
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
errno: a standard unix feature. The numbers are platform dependent.
I
would assume in your case
ETIMEDOUT 110 /* Connection timed out */
so no wonder, that's exactly what we expect (and doesn't tell us
the
reason, i.e. what's wrong on the *backend* taking that long for a
response).
-> receiving reply from tomcat failed with out recovery in send
loop
attempt=0 - ? with out recovery in send loop - means?
That your configuration doesn't allow us to send the request to
another
backend. recovery_options 7 include: if mod_jk was able to send the
request to a backend, do not try to send it to another backend in
case
of an error during the response handling. Even if you would allow
sending to another backend, it would not help with *not* putting
the
worker into error state. More likely would be, that you would put
all
workers into error state, because all of them might run into the
same
timeout, one after the other.
-> unrecoverable error 504 - details to this error ?
That's simply how we return the situation back to the client
(browser).
Ok - i turn the logging level to debug - the course of events get
more
clear - but also more questions appear - there are socket numbers
-
which sockets - what are these numbers e.g will be shutting down
socket
35 for worker INETP1021 - The sockets are good for ? - how many
are
there/per worker ? can i configure them ?
Should not be the problem here. For apache httpd if you do *not*
configure anything, we automatically choose the number of httpd
threads
as the maximum number of connections. No need to change anything
here.
=> Generally -How can i solve such problems - i tried to look into
the
mod_jk code - searching for error codes, error messages - but
cannot
find some relevant informations, - i am studying the log Files -
but
don't find out what really happens.
Post to the list. Improve our dics.
The error message contains the word "timeout" and "reply" and you
have
a
"reply_timeout".
Long running requests are a frequent problem. If you want to get
rid of
them, start by adding response times to your httpd and your tomcat
access log format (%D). Then have a look, which URLs are producing
long
running requests, during what time of day are they happening etc.
This
might give you a clue about the reasons.
And if they are very frequent: do Java Thread Dumps of your
backends
and
analyze them.
So - maybe someone has an idea why the worker think that the
corresponding Tomcat is dead, and why he will not recover by
itself. !
Tomecat is dead: from the point of view of mod_jk it simply means:
we
didn't get an answer, when we expected one. Details depend on the
additional log lines (could not connect, reply timeout etc.).
And i am also searching for tips how i can help myself - and where
to
find something about the error codes, messages,..in mod_jk
thanks for your attention
Best
ahmed musa (writing from vienna)
Regards,
Rainer
Current Infrastructur
We have 3 Apache Webserver (2.2.6) -based on CentOS release 4.3
/Kernelversion 2.6.9-34
In front of the Webserver there are two (two Locations)
HW-Loadbalancer
(but they have no role in this story)
The Webservers are hosted at our ISP.
The Webserver balance the requests via mod_jk (Version 1.2.25) for
approx. 10 Webapps to 18 Backend-Tomcatserver (Bladeserver -
because
of
underlying Application-Parts the OS is Windows 2003 Server - a
long
story not worth to explain :-) ). The Tomcatserver gain Data via
Requests against DB2 Server/DB2-Databases on the Mainframe. The
Tomcatserver are Inhouse -and were rebooted nightly because of
automated
Deployment processes.
Between the Webserver and the Tomcatserver is a Checkpoint
Firewall.
All webapps are deployed on all Tomcats - only mod_jk manages the
requests to certain Tomcat- instances.
(on one Bladeserver there are two identically Tomcat Instances
running).
Versions: Tomcat - 5.5.17_11, JDK 1.5.0_11-b03. The requests
against
the public Website(s) are normal short living requests - not many
-
The
most Webapps (Portals) need a login, have a strong focus on
business
logic - so the instances are big (many MBs in RAM), the sessions
are
sticky and the session timeout is 20 minutes. But there are also
less
requests. To the User requests - Monitoring requests from our ISP
are
added.
The Problems appears at Servers/Portals which very less
Userrequests.
worker.properties
worker.list=ajp_bam,ajp_ggi,ajp_ad,ajp_svp,.......,jkstatus
worker.template.type=ajp13
worker.template.lbfactor=5
worker.template.socket_keepalive=1
worker.template.connect_timeout=7000
worker.template.prepost_timeout=5000
worker.template.reply_timeout=120000
worker.template.retries=6
worker.template.activation=Active
worker.template.recovery_options=7
worker.lbtemplate.type=lb
worker.lbtemplate.max_reply_timeouts=6
worker.lbtemplate.method=Session
#Produktions Worker
# AS-INETP101 - 106 - 6/6 GGI
worker.INETP1011.host=AS-INETP101.AEAT.ALLIANZ.AT
worker.INETP1011.port=65001
worker.INETP1011.reference=worker.template
....many more of the same
then
worker.ajp_ad.reference=worker.lbtemplate
worker.ajp_ad.balance_workers=INETP1032,INETP1062
.... many more portals
at least jkstatus
The JKMount is very simple
JkMount /* ajp_ad --- for the other portals mostly the same
The Portals are Virtual Hosts on the Apache.
Tomcat - server.xml
example
<Connector port="65001" maxThreads="300" protocol="AJP/1.3" />
<Engine name="Catalina" jvmRoute="INETP5021"
defaultHost="default">
......
<Host name="slfinsol.com" appBase="webapps" unpackWARs="true"
autoDeploy="false" deployOnStartup="false" xmlValidation="false"
xmlNamespaceAware="false">
<Alias>www.slfinsol.com</Alias>
<Alias>web1.slfinsol.com</Alias>
...
<Alias>testweb.slfinsol.com</Alias>
.....
<Valve
className="org.apache.catalina.valves.AccessLogValve"
directory="logs" prefix="swl_access_log." suffix=".txt"
pattern="common"
resolveHosts="false" />
<Valve
className="at.allianz.tomcat.valve.RequestTimeValve"/>
<Valve
className="at.allianz.tomcat.valve.WebcollaborationWorkaroundValve"/>
<Context path="" docBase="swl" />
<Context path="/monitor5" docBase="monitor" />
<Context path="/swl" docBase="swl" />
</Host>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
