Re: Tomcat, REMOTE_USER, getRemoteUser()
> I haven't looked too closely, but I'm not sure what "standard" > mechanisms there are to communicate this through a proxy. variables > don't pass through a proxy, and a HEADER is NOT the proper solution here > unless you also implement something similar to the Tomcat RemoteIpValve > where you have the notion of trustedProxiesForAuth or something like that. Neither AJP forwarding REMOTE_USER or an HTTP header is great, so if we all care about security, that feature of mod_jk needs disabling with warnings/sirens should one enable it. I do appreciate the remote IP valve exists, but this is a sticking plaster around the core design flaw. However, it is true that plenty of vendor modules exist in the Apache HTTPD world that forward a username on a header (I've listed some) and with the appropriate controls in place, it isn't an awful solution to use an HTTP header to carry the username. It's no different to mod_jk forwarding REMOTE_USER (mod_jk isn't providing security in our puzzle). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat, REMOTE_USER, getRemoteUser()
Hello,
> I'm not sure how long ago that was, but I don't live in the Windows
> world. I would have thought that someone at Apache Lounge would have
> balked if a release was broken. Were you building a release version,
> or trunk?
I downloaded a release. This was a few years ago now. I suspect mod_jk
on Windows is no longer well used. Windows as a hosting platform in
larger organisations (the type that use Tomcat) is not common, and
vendors ship some random old version of the module dating back years
("it works [*], why change it?").
[*] we don't do security.
> > And having got involved and contributed a few years ago, I recall
> > the code being somewhat in need of refurbishment, as does pretty
> > much all 15+ year old C/Java/
>
> Last time I checked, C code pretty much compiled the same way it did
> 25 years ago. I don't see a particular need to go in there and change
> a whole lot of stuff.
Just because it compiles doesn't mean it's a work of art. I look at much
of what I wrote ten years ago, back in the Tomcat 3 days, and cry.
> Nobody is making anyone use AJP.
They are encouraging it if it's doing something the HTTP connector does
not.
> be invented. Probably some meta-information passed-through in HTTP
> headers. It would work, but it would be a pain, and it would be more
> difficult to hide those implementation details from the application.
Disagree. It's entirely easy to publish the username of common SSO
systems (Oracle Access Manager, OpenSSO/AM, Tivoli, SiteMinder, RSA
Access Manager [ClearTrust] to name but a few) as an HTTP header. The
typical integration approach is "publish username as an HTTP header
using Apache module, reverse proxy onto Tomcat and pick up via your Java
app".
I'm suggesting that this support should be more integrated to the
getRemoteUser() call, as has been implemented with AJP.
> >> So, your proposed implementation is incorrect and represents a
> >> security vulnerability.
> >
> > It does not represent a security vulnerability.
>
> Sure it does: a client can supply a forged REMOTE_USER header quite
> easily.
Yes, and that's precisely the problem with mod_jk. Anyone with local
access to the host can do the same as a
Apache/mod_proxy_http/REMOTE_USER solution. I'm amazed the wider
security world hasn't picked up on the widely abused mod_jk security
hole.
> So, how is Tomcat supposed to know that the request has been
> properly-sanitized? At least with AJP, one expects that a nearby web
> server is making the request and that the user knows what they are
> doing. Nobody sets up a one-box-wonder in production with the AJP port
> publicly available. But people do that all the time with port 80
> available publicly.
Well, for a start, I'm not sure the AJP connector comes bound to
localhost (tomcat 8.x download):
Next, no-one makes any effort to secure AJP to Apache. And whilst I'm
sympathetic to the sanitisation of HTTP headers (by Apache) belief, I'm
not sure Tomcat's now decade old HTTP connector - that has had lots of
attention/work - needs the protection once offered by Apache HTTPd. It
would certainly be interesting to get the two pen-tested.
The sad fact is that the majority of Tomcat AJP enabled deployments with
tomcatAuthentication="false" are vulnerable to injection, perhaps even
not from localhost!
> Care to file a Bugzilla enhancement for that?
How do? :)
> I don't like your proposed implementation, but that doesn't mean that
> it could work similar to the way you've described it. For one thing,
> it should be a configuration option that is off *by default*.
Actually, I don't like it very much, but I'd like to see some
consistency. This includes extensive logging/warning/nagging when AJP is
not bound to localhost and tomcatAuthentication="false", and hence
greater transparency on this security issue.
I'd also like a better way and after discussing with some
security-geeks, we were wondering if there's some way we can implement a
Valve that takes a username and a signature using a shared secret. The
problem is signing in Apache: I've not looked too hard for a module to
do this but maybe one exists? If one does exist, then the mod_jk module
could use the same strategy to ensure Tomcat only trusts a username +
valid signature.
John
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat, REMOTE_USER, getRemoteUser()
> BlueCothe protocol that one can't get away from; that everyone > understands and can easily debug with telnet/etc)at, etc. Sorry, I think my brain started to fail. "Blue Coat; HTTP is a protocol that everyone understands / can easily debug with telnet / etc." - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat, REMOTE_USER, getRemoteUser()
Hello, > Anything in particular? Plumbing code is always not terribly pretty. > > It's kept up-to-date and generally supports more features than > mod_proxy_ajp. Well, there was a point where 64bit windows builds didn't even work - which tells me there's not a lot of testing going on. And having got involved and contributed a few years ago, I recall the code being somewhat in need of refurbishment, as does pretty much all 15+ year old C/Java/ > > I see your point regarding the ajp protocol but equally, HTTP is > > everywhere and if every other part of a web stack is HTTP, there > > seems little value in doing anything different between Apache & > > Tomcat. > > There are some slight optimizations in comparison to AJP. When HTTPS/2 > (which is very much like AJP) hits, you'll see less of a difference. I don't think there's any good reason to use AJP over HTTP with respect to web performance. Pretty much any bottleneck exists in the application, or the plethora of components it uses, not the transport layer. AJP is merely obscuring a request that in many (commercially deployed) solutions originated from a device with a number of other solutions in-between it and Apache/IIS, ie the F5, an Akami layer, BlueCothe protocol that one can't get away from; that everyone understands and can easily debug with telnet/etc)at, etc. > > AJP has various load balancing features/etc and if that's what one > > wants, fine, but most organisations have hardware load balancers > > etc to do this for them now-a-days. > > Also, mod_proxy_* does load-balancing. This is not a feature of the > protocol, but of the tool. Yes, I appreciate that, but mod_proxy/mod_jk LB is not used in your day to day commercial solution that - rightly or wrongly - consider Apache/Tomcat as one (often, people don't even know the difference!) and do the load balancing in the corporate wide solution. > > Going back to my request, I note the Servlet Specification API > > docs state that getRemoteUser should return the CGI variable > > REMOTE_USER: > > > > http://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/ > HttpServletRequest.html#getRemoteUser() > > > > But as I've highlighted, it does not, so Tomcat is in breach of > > the spec. How can this be raised as a bug? > > REMOTE_USER is not an HTTP header... it's the "REMOTE_USER" CGI > variable Yet they are used interchangeably. And we're not discussing mod_[fast]cgi, but a completely different architecture/solution. > So, your proposed implementation is incorrect and represents > a security vulnerability. It does not represent a security vulnerability. Any application that trusts a user via REMOTE_USER, whether via the HTTP or AJP connector is doing so knowingly. And in such a case, a pen-test would fail unless there were extra measures in place (Tomcat's AJP or HTTP connector listening on localhost; restricted access to the host), but they are no substitute for authenticating within Tomcat. Apache authenticating on behalf of Tomcat is a bad decision, but a decision still forced upon some because of vendors not providing Java authentication solutions out of the box (ie CA SiteMinder, RSA Access Manger, Tivoli, etc). If the vendor pushes and only actively supports an Apache front end, one has to use an Apache front end. Setting up mod_proxy_http is easier than mod_[proxy_]jk. Everyone understands HTTP, and I'd like Tomcat to more easily integrate with Apache (and the various authentication modules) using HTTP. > Tomcat is compliant insofar as there is no standard that covers where > the REMOTE_USER CGI variable should be set or what its value should > be. When a user is logged-into a web application, > request.getRemoteUser returns the name of the currently-logged-in > user. When no user is logged-in, it returns null. That seems to be > compliant to me. Well, admitting there's no standard and then stating is compliant is an interesting conclusion :) > How do you want it to behave instead? It should be consistent with the AJP Interface. Either they both return a user on getRemoteUser() or they don't. The tomcatAuthentication flag on the AJP connector tells Tomcat to trust the REMOTE_USER header/CGI variable/whatever we wish to call it. With a simple change to the HTTP connector, it can do the same. Rename REMOTE_USER to X_UNTRUSTED_USER if you wish, but this no more or no less secure than the AJP connector because one has to 'opt in' to the security issue. I would also like to see Tomcat reporting the obvious security hole when this approach is used (for either AJP or HTTP) by reporting to the logs if the connector is not bound to localhost/etc. I'm amazed at how often I find Apache+Tomcat deployed with zero security on the AJP connector and no-one appearing to understand the issue. > If you want to have the web server report the REMOTE_USER CGI variable > to Tomcat, you'll have to arrange that for yourself. And indeed it's not hard to do so, but if one wa
Re: Tomcat, REMOTE_USER, getRemoteUser()
I agree that mod_proxy_ajp is more commonly supported. I've looked/briefly worked on the mod_jk source and it was pretty awful - but that's what happens to code over time. I see your point regarding the ajp protocol but equally, HTTP is everywhere and if every other part of a web stack is HTTP, there seems little value in doing anything different between Apache & Tomcat. AJP has various load balancing features/etc and if that's what one wants, fine, but most organisations have hardware load balancers etc to do this for them now-a-days. Going back to my request, I note the Servlet Specification API docs state that getRemoteUser should return the CGI variable REMOTE_USER: http://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/http/HttpServletRequest.html#getRemoteUser() But as I've highlighted, it does not, so Tomcat is in breach of the spec. How can this be raised as a bug? On Fri, Jul 24, 2015, at 09:39 AM, S.Booth wrote: > On 23/07/15 20:38, John Baker wrote: > > The flag to which you refer is for AJP only, hence the inconsistency (as > > AJP becomes less common and reverse proxying HTTP becomes the norm). > > While I agree with you that the http connector should be consistent with > the AJP behavior where possible I'm interested if you think reverse > proxy http is better for any reason. I switched to mod_proxy_ajp from > mod_jk because it seemed more commonly supported out of the box but I > never saw any reason to give up on ajp. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat, REMOTE_USER, getRemoteUser()
Sorry, I was reading the latest Tomcat 8.0.24 source to verify how it
worked (it hasn't changed in some time it seems).
The flag to which you refer is for AJP only, hence the inconsistency (as
AJP becomes less common and reverse proxying HTTP becomes the norm).
On Thu, Jul 23, 2015, at 08:05 PM, Violeta Georgieva wrote:
> Hi,
>
> 2015-07-23 21:54 GMT+03:00 John Baker :
> >
> > Hello,
> >
> > I note the HTTP connector does the following when
> > Request.getRemoteUser() is called:
> >
> > public String getRemoteUser() {
> > if (userPrincipal == null) {
> > return null;
> > }
> > return userPrincipal.getName();
> > }
> >
> > I understand what it's trying to do but it's not consistent with the AJP
> > adapter and doesn't make it helpful to interact with upstream reverse
> > proxies setting the username as a header (ie REMOTE_USER). Would it be
> > possible to change the implementation to this:
> >
> > public String getRemoteUser() {
> > String user = userPrincipal.getName();
> > if (userPrincipal == null)
> > user= getHeader("REMOTE_USER");
> > return user;
> > }
> >
> > Or even better, allow the REMOTE_USER header name to be set via a
> > configuration value? This seems much more consistent with the AJP
> > adapter that does pick up REMOTE_USER (sent by the ajp module) and
> > avoids users having to retrieve a header from the request.
> >
>
> You didn't specify any Tomcat version.
>
> Check this feature [1] -> tomcatAuthentication/tomcatAuthorization
>
> Regards,
> Violeta
> [1]
> http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html#Standard_Implementations
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat, REMOTE_USER, getRemoteUser()
Hello,
I note the HTTP connector does the following when
Request.getRemoteUser() is called:
public String getRemoteUser() {
if (userPrincipal == null) {
return null;
}
return userPrincipal.getName();
}
I understand what it's trying to do but it's not consistent with the AJP
adapter and doesn't make it helpful to interact with upstream reverse
proxies setting the username as a header (ie REMOTE_USER). Would it be
possible to change the implementation to this:
public String getRemoteUser() {
String user = userPrincipal.getName();
if (userPrincipal == null)
user= getHeader("REMOTE_USER");
return user;
}
Or even better, allow the REMOTE_USER header name to be set via a
configuration value? This seems much more consistent with the AJP
adapter that does pick up REMOTE_USER (sent by the ajp module) and
avoids users having to retrieve a header from the request.
Thanks
John
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: confirm subscribe to users@tomcat.apache.org
m > (out1-smtp.messagingengine.com [66.111.4.25]) > by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) > with ESMTPS id 045D043DD9 > for ; Thu, 23 Jul 2015 18:46:52 > + (UTC) > Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) > by mailout.nyi.internal (Postfix) with ESMTP id 02FAC2024A > for ; Thu, 23 Jul 2015 14:42:38 > -0400 (EDT) > Received: from frontend2 ([10.202.2.161]) > by compute5.internal (MEProxy); Thu, 23 Jul 2015 14:42:38 -0400 > DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= > messagingengine.com; h=content-transfer-encoding:content-type > :date:from:message-id:mime-version:to:x-sasl-enc:x-sasl-enc; s= > smtpout; bh=2jmj7l5rSw0yVb/vlWAYkK/YBwk=; b=m27qo6GK9j0GR0sMCdHH > O6OmEsduURVV7vYF9qFW6KI0koFloRE9xuFsaXV+JpqdHH6eRsMFJGhyR+8X9bqz > LXHG8age9quYm84NZqPsyHow40pUXbb9qk8ADJB2Vrr4iWSpOJzqsIj16TJQJzJK > XTWz0Xf3Z4QeRe/vQ5l4ye8= > X-Sasl-enc: QiqmsBd7kaFR3jbsBkPNDfvpWB+1YyxoLcLbeinq/UNP 1437676957 > Received: from [192.168.0.2] (jbaker1976.plus.com [80.229.26.4]) > by mail.messagingengine.com (Postfix) with ESMTPA id 9DDC26800C7 > for ; Thu, 23 Jul 2015 14:42:37 > -0400 (EDT) > Message-ID: <55b1359e.1090...@dryfish.org.uk> > Date: Thu, 23 Jul 2015 19:42:38 +0100 > From: John Baker > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 > Thunderbird/31.5.0 > MIME-Version: 1.0 > To: users-subscr...@tomcat.apache.org > Content-Type: text/plain; charset=utf-8; format=flowed > Content-Transfer-Encoding: 7bit > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat GzipOutputFilter
The AJP connector doesn't support compression (a missing useful feature in my opinion) so I found myself looking for a filter. Thanks for letting me know it won't work. I'll look at the example filter. On Fri, 01 Apr 2011 16:28 +0400, "Konstantin Kolinko" wrote: > 2011/4/1 John Baker : > > Hello, > > > > I note there's a GzipOutputFilter in the Tomcat (and JBoss) jar files: > > > > http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/coyote/http11/filters/GzipOutputFilter.html > > > > Yet I can't load it in the web.xml file using . Is there > > some pre-defined filter name for this filter? > > This class does not implement the javax.servlet.Filter interface and > thus it cannot be used in a web application. > > It is just an internal component of HTTP connectors, that is used when > you enable gzip compression on the connector, > http://tomcat.apache.org/tomcat-6.0-doc/config/http.html > > There is "compressionFilters.CompressionFilter" class in the examples > webapp though, that can be used as a Filter. > > Best regards, > Konstantin Kolinko > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat GzipOutputFilter
Hello, I note there's a GzipOutputFilter in the Tomcat (and JBoss) jar files: http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/coyote/http11/filters/GzipOutputFilter.html Yet I can't load it in the web.xml file using . Is there some pre-defined filter name for this filter? John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk / AJP and compression
Mark, In general, that's true. However, for high performance applications measuring results in milliseconds - ie noting that a request with compression takes 60ms and without takes 120ms - it's an issue. And it seems so easy to fix by putting compression into the Tomcat AJP connector. Given it's a quick fix, I was curious to why it hadn't been done before now? John On Fri, 01 Apr 2011 11:20 +0100, "Mark Thomas" wrote: > On 01/04/2011 11:15, John Baker wrote: > > Hello, > > > > The Tomcat AJP Connector does not support compression. Why has this been > > ommitted? > > Because the link between Tomcat and reverse proxy will nearly always > have significantly more capacity than the link between the client and > the reverse proxy. Adding compression gains very little but adds delay > and increases CPU usage. > > Mark > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
mod_jk / AJP and compression
Hello, The Tomcat AJP Connector does not support compression. Why has this been ommitted? Of course, Apache can do compression but this leaves data running uncompressed between Apache and Tomcat. John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Hello, I've spent a week looking into mod_jk, Apache, etc., and the problem appears to be due to a bug in the AJP connector within JBoss: https://jira.jboss.org/browse/JBPAPP-366 This bug is not fixed in the latest release of JBoss 4.2.3, which is rather disappointing. After downloading the jbossweb-2.0.1-GA source from subversion, patching, recompiling with Java 1.5 (if you want it to work on the Java 1.5 and 1.6 versions of JBoss 4.2.3), the problem disappears. While solving this problem, I identified the need for a debug statement to be generated from mod_jk when the poll(..) fails, i.e. waited 2 seconds and discovered the socket was not being shut properly. Mladen has kindly added this to the next release. Thanks to everyone who's contributed and if I have any further updates, I will let you know. John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Thursday 09 September 2010 16:08:04 you wrote: > On 09/09/2010 04:48 PM, John Baker wrote: > > > > > > On Thursday 09 September 2010 15:45:44 you wrote: > >> Nice. > > > > I spoke too soon. I'm now trying to figure out how to print out the IP > > address of the socket (I don't really do C) so I can log the socket that > > caused the poll to timeout, and compare with a tcpdump. > > > > Try with getsockname, but it's not a trivial task. > > Perhaps using /proc/pid/fd > and matching the socket# we have with the actual info > eg. > 32 -> socket:[port] Isn't the address stored with jk_sock_t so I can use jk_dump_hinfo, as I can see is in the open socket method? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Interestingly, some of our JBoss instances are showing a large number ajp threads that seem to be in keep alive mode but are well beyond the connectionTimeout defined in server.xml (which is set to 9): Max threads: 40 Current thread count: 40 Current thread busy: 40 Max processing time: 5563 ms Processing time: 2552.538 s Request count: 75522 Error count: 6 Bytes received: 0.00 MB Bytes sent: 879.42 MB Stage TimeB Sent B Recv Client VHost Request K 508896 ms ? ? 10.113.168.4? ? K 93386 ms? ? 10.113.168.4? ? K 7147 ms ? ? 10.113.168.4? ? K 3994 ms ? ? 10.113.168.4? ? K 660 ms ? ? 10.113.168.4? ? K 735822 ms ? ? 10.113.168.4? ? K 71565 ms? ? 10.113.168.4? ? Almost as if connectionTimeout is being ignored in some cases - this is JBoss 4.2.3, so an oldish version of Tomcat. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
I would be happy to share all my evidence and write a report once we get to the bottom of this problem. Any hints on printing out the socket IP (i.e. of Tomcat)? On Thursday 09 September 2010 15:47:33 you wrote: > On 09/09/2010 03:22 PM, John Baker wrote: > > > > Do you fancy putting that change into the next release of mod_jk? > > > > BTW, do you fancy opening BZ report fill in what we > tried so far and the actual solution, so we can track that down > for the next releases and patches? > > Regards - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Thursday 09 September 2010 15:45:44 you wrote: > Nice. I spoke too soon. I'm now trying to figure out how to print out the IP address of the socket (I don't really do C) so I can log the socket that caused the poll to timeout, and compare with a tcpdump. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Thursday 09 September 2010 13:59:50 you wrote: > > > > If that doesn't help, it's obvious the Tomcat > > doesn't close the socket, so should be investigated > > why. Like said before, either the Tomcat doesn't > > respond to shutdown or the shutdown's FIN packet > > isn't send to the Tomcat or back to mod_jk, > > due to some weird OS networking issues. > > I think there's been rather a lot of speculation on this problem :-) The change to fds.events appears to be working! Thanks. I'm somewhat surprised this hasn't been picked up anytime before? We can't be the only people deploying hundreds of worker threads/JBoss instances, and there are no firewalls between mod_jk/JBoss. I guess it's possible the AJP thread on one sever was 'stuck'. Do you fancy putting that change into the next release of mod_jk? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Thanks for the feedback.
Can you tell me why this if statement exists:
if (poll(&fds, 1, timeout) > 0)
{
...
}
else
break;
It appears to be at fault.
John
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.0 which gets stuck
I thought an illegal state exception would occur if an attempt was made to write to a socket that's now shut but you are right, it looks more like an attempt to perform an internal redirect after response has been committed (although that message should appear in the logs). -Original Message- From: Maximilian Stocker Date: Wed, 8 Sep 2010 12:43:01 To: 'Tomcat Users List' Reply-To: "Tomcat Users List" Subject: RE: Tomcat 5.0 which gets stuck I don't know what you mean by "We are using Apache Tomcat 5.0 which gets stuck sometimes and needs a stop and restart" but the stack trace you posted is almost certainly evidence of a bug in your code. It's an illegal state exception to try and send an error if you have already sent "normal" content in some way. Which you almost certainly have. So the stack trace shows a bug in your code. But what do you mean that tomcat "gets stuck"? What exact symptoms does this display. As to the other poster I am not sure what sort of guess that was but there is nothing in that stack trace relating to broken pipes, timeouts, broken sockets etc. This is an HTTP protocol error (caused by poorly written code) not a TCP level error. -Original Message- From: Sumeet Chitte [mailto:chittesum...@gmail.com] Sent: Wednesday, September 08, 2010 12:19 PM To: users@tomcat.apache.org Subject: Fwd: Tomcat 5.0 which gets stuck Hi, We are using Apache Tomcat 5.0 which gets stuck sometimes and needs a stop and restart. We are using Microsoft Windows 2000 Service Pack 4. Below is the error from the log file, Kindly provide some solution. 2010-09-07 10:33:38 StandardWrapperValve[IFSCoreServlet]: Servlet.service() for servlet IFSCoreServlet threw exception java.lang.IllegalStateException at org.apache.coyote.tomcat5.CoyoteResponseFacade.sendError(CoyoteResponseFacade.java:324) at com.infodata.ifs.core.IFSCoreServlet.sendError(IFSCoreServlet.java:1159) at com.infodata.ifs.core.IFSCoreServlet.service(IFSCoreServlet.java:1057) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) -- Thanks & Regards, Sumeet G. Chitte | Software Engineer, IBM,EGL 'A' Block, Off Indiranagar, Kormagala Intermediate Ring Road, Bangalore : 560071 Mail: suchi...@in.ibm.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: Tomcat 5.0 which gets stuck
Looks like your application took too long to respond and by the time it tried to write to the output stream, it had been closed. Have you set any connection/socket timeouts? -Original Message- From: Sumeet Chitte Date: Wed, 8 Sep 2010 21:48:31 To: Reply-To: "Tomcat Users List" Subject: Fwd: Tomcat 5.0 which gets stuck Hi, We are using Apache Tomcat 5.0 which gets stuck sometimes and needs a stop and restart. We are using Microsoft Windows 2000 Service Pack 4. Below is the error from the log file, Kindly provide some solution. 2010-09-07 10:33:38 StandardWrapperValve[IFSCoreServlet]: Servlet.service() for servlet IFSCoreServlet threw exception java.lang.IllegalStateException at org.apache.coyote.tomcat5.CoyoteResponseFacade.sendError(CoyoteResponseFacade.java:324) at com.infodata.ifs.core.IFSCoreServlet.sendError(IFSCoreServlet.java:1159) at com.infodata.ifs.core.IFSCoreServlet.service(IFSCoreServlet.java:1057) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) -- Thanks & Regards, Sumeet G. Chitte | Software Engineer, IBM,EGL 'A' Block, Off Indiranagar, Kormagala Intermediate Ring Road, Bangalore : 560071 Mail: suchi...@in.ibm.com
Re: 2 second delays in mod_jk while "maintaining workers"
I think it would be helpful if you could walk us through the code. The shutdown function provides three error codes - the jk function doesn't check them, and the most likely is 'socket closed' (ie jboss closed it?), hence I'm unsure what the drain code is doing. --Original Message------ From: John Baker To: Tomcat Users List ReplyTo: Tomcat Users List Subject: Re: 2 second delays in mod_jk while "maintaining workers" Sent: 8 Sep 2010 16:41 > On 09/08/2010 05:08 PM, John Baker wrote: > The code *is* required. > It is used when the client disconnects while the backend > still has some data in the AJP buffer. Drain is needed > to read that excess data. Why does it always report 0 bytes read? > If you can compile mod_jk and test, try adding some debug messages inside > the poll loop. Anyhow, seems to me there is some sort of network > problem you are facing. That's next on my list. > Have you been able to see the traffic using Wireshark? Not yet. > > > Regards > -- > ^TM > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
> On 09/08/2010 05:08 PM, John Baker wrote: > The code *is* required. > It is used when the client disconnects while the backend > still has some data in the AJP buffer. Drain is needed > to read that excess data. Why does it always report 0 bytes read? > If you can compile mod_jk and test, try adding some debug messages inside > the poll loop. Anyhow, seems to me there is some sort of network > problem you are facing. That's next on my list. > Have you been able to see the traffic using Wireshark? Not yet. > > > Regards > -- > ^TM > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Hello,
I don't thikn the shutdown call is to blame - I think it's the large pile
of code below. i.e. everything below the if (shutdown(..)) below. The
question is, what does it all do and does it actually work? It appears to
be the 'drain' code, but given it often results in this message:
"Shutdown socket .. and read 0 lingering bytes in 2 sec."
then it would suggest it doesn't work. The comments also suggest the 2
seconds is "guess work".
I've sent SECONDS_TO_LINGER and MAX_SECS_TO_LINGER to 0 and the problem
goes away, and I've seen this discussed elsewhere on the forum.
So, if this code isn't needed, why is it still there? And if it is
needed, why does it seem not to work and what is the correct figure for
SECONDS_TO_LINGER? If this is being performed inline then I'd suggest 2s
is far too long - clients don't want to wait 2s for a request to complete
(ideally, socket clean up should not be done in this thread if any linger
is required).
John
..
MAX_SECS_TO_LINGER= 30
SECONDS_TO_LINGER= 2
...
int jk_shutdown_socket(jk_sock_t sd, jk_logger_t *l)
{
...
/* Shut down the socket for write, which will send a FIN
* to the peer.
*/
if (shutdown(sd, SHUT_WR)) {
rc = jk_close_socket(sd, l);
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_DEBUG, "Failed sending SHUT_WR for socket
%d", sd);
errno = save_errno;
JK_TRACE_EXIT(l);
return rc;
}
do {
#ifdef HAVE_POLL
struct pollfd fds;
fds.fd = sd;
fds.events = POLLIN;
#else
fd_set rs;
FD_ZERO(&rs);
/* Read all data from the peer until we reach "end-of-file"
* (FIN from peer) or we've exceeded our overall timeout. If the
* backend does not send us bytes within 2 seconds
* (a value pulled from Apache 1.3 which seems to work well),
* close the connection.
*/
FD_SET(sd, &rs);
tv.tv_sec = timeout / 1000;
tv.tv_usec = (timeout % 1000) * 1000;
#endif
rp = 0;
#ifdef HAVE_POLL
if (poll(&fds, 1, timeout) > 0)
#else
if (select((int)sd + 1, &rs, NULL, NULL, &tv) > 0)
#endif
{
do {
#if defined(WIN32) || (defined(NETWARE) && defined(__NOVELL_LIBC__))
rc = recv(sd, &dummy[0], sizeof(dummy), 0);
if (JK_IS_SOCKET_ERROR(rc))
JK_GET_SOCKET_ERRNO();
#else
rc = read(sd, &dummy[0], sizeof(dummy));
#endif
if (rc > 0)
rp += rc;
} while (JK_IS_SOCKET_ERROR(rc) && (errno == EINTR || errno ==
EAGAIN));
if (rc < 0)
break;
}
else
break;
rd += rp;
if (rp < sizeof(dummy)) {
if (timeout > SECONDS_TO_LINGER) {
/* Try once again with 1000 times smaller timeout
* In our case 2 msec.
*/
timeout = SECONDS_TO_LINGER;
continue;
}
/* We have read less then size of buffer
* It's a good chance there will be no more data
* to read.
*/
if ((rc = sononblock(sd))) {
rc = jk_close_socket(sd, l);
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_DEBUG,
"error setting socket %d to nonblocking", sd);
errno = save_errno;
JK_TRACE_EXIT(l);
return rc;
}
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_DEBUG,
"shutting down the read side of socket %d", sd);
shutdown(sd, SHUT_RD);
break;
}
timeout = SECONDS_TO_LINGER * 1000;
} while (difftime(time(NULL), start) < MAX_SECS_TO_LINGER);
rc = jk_close_socket(sd, l);
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_DEBUG,
"Shutdown socket %d and read %d lingering bytes in %d sec.",
sd, rd, (int)difftime(time(NULL), start));
errno = save_errno;
JK_TRACE_EXIT(l);
return rc;
}
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Hello
We've just noticed that the maintenance mode operates on all workers, so
having one worker run maintenance for the rest is making tracing the
problem difficult. Reading down the logs, we can see it finds a worker
and iterates through workers perforing maintenance.
Is this correct?
Moving on, consider the following debug:
[Wed Sep 08 10:36:16.907 2010] [2205:47701852022112] [trace]
ajp_abort_endpoint::jk_ajp_common.c (739): enter
[Wed Sep 08 10:36:16.907 2010] [2205:47701852022112] [trace]
jk_shutdown_socket::jk_connect.c (713): enter
[Wed Sep 08 10:36:16.907 2010] [2205:47701852022112] [debug]
jk_shutdown_socket::jk_connect.c (722): About to shutdown socket
The next entry in the log with [2205:47701852022112] is the following:
[Wed Sep 08 10:36:18.907 2010] [2205:47701852022112] [trace]
jk_close_socket::jk_connect.c (650): enter
[Wed Sep 08 10:36:18.908 2010] [2205:47701852022112] [trace]
jk_close_socket::jk_connect.c (665): exit
[Wed Sep 08 10:36:18.908 2010] [2205:47701852022112] [debug]
jk_shutdown_socket::jk_connect.c (813): Shutdown socket 156 and read 0
lingering bytes in 2 sec.
[Wed Sep 08 10:36:18.908 2010] [2205:47701852022112] [trace]
jk_shutdown_socket::jk_connect.c (817): exit
Looking at the code in jk_connect.c:
int jk_shutdown_socket(jk_sock_t sd, jk_logger_t *l)
{
...
if (JK_IS_DEBUG_LEVEL(l))
jk_log(l, JK_LOG_DEBUG, "About to shutdown socket %d", sd);
/* Shut down the socket for write, which will send a FIN
* to the peer.
*/
if (shutdown(sd, SHUT_WR)) {
rc = jk_close_socket(sd, l);
And the jk_close_socket function:
int jk_close_socket(jk_sock_t sd, jk_logger_t *l)
{
int rc;
int save_errno;
JK_TRACE_ENTER(l);
Tieing up with the logs suggests the shutdown(sd, SHUT_WR) method is
blocking.
Any thoughts,
John
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Where can I find documentation on JkWatchdog? On Tuesday 07 September 2010 13:51:23 you wrote: > - As a workaround: using a JkWatchdog moves the maintain into a separate > thread. But during the socket closing a lock is held, which blocks other > threads from accessing the same worker. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: > I don't like "socket_timeout" ... > > > worker.basic.socket_timeout=90 > > but I would like socket_connect_timeout. > > The next two are possibly a bit short, because if the backend e.g. does > a Java Garbage Collection which miht take longer than 1 second, tose > timeouts will fire and take the node out of load balancing. > > > worker.basic.connect_timeout=1000 > > worker.basic.prepost_timeout=1000 I've change socket_timeout=90 to socket_connect_timeout=9000, and connect_timeout/prepost_timeout to 5000, and the problem persists: [Tue Sep 07 11:29:22.863 2010] [364:47921619424608] [debug] jk_shutdown_socket::jk_connect.c (722): About to shutdown socket 152 [Tue Sep 07 11:29:24.863 2010] [364:47921619424608] [debug] jk_shutdown_socket::jk_connect.c (813): Shutdown socket 152 and read 0 lingering bytes in 2 sec. [Tue Sep 07 11:29:24.868 2010] [364:47921619424608] [debug] wc_maintain::jk_worker.c (339): Maintaining worker lb-jboss51-integration John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Tuesday 07 September 2010 11:13:07 you wrote: > It's obvious that > > shutdown(socket, SHUT_WR) > poll(socket, 2 seconds) > close(socket) > > caused poll call to time out, meaning that > the JBoss side didn't respond to the > shutdown(socket, SHUT_WR) call by > closing it's side of the connection. > > BTW, do you have "lingering bytes in 0 sec" messages > in your log or lower then 2 sec? Yes. There's plenty of lingering 0 sec. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: > > [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] > > find_match::jk_uri_worker_map.c (850): Attempting to map context URI > > '/*=lb-jboss51-integration' source 'JkMount' > > [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] > > find_match::jk_uri_worker_map.c (863): Found a wildchar match > > '/*=lb-jboss51-integration' > > [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] > > jk_handler::mod_jk.c (2462): Into handler jakarta-servlet > > worker=lb-jboss51-integration r->proxyreq=0 > > [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] > > wc_get_worker_for_name::jk_worker.c (116): found a worker > > lb-jboss51-integration > > [Tue Sep 07 10:20:20.618 2010] [18806:46962404156768] [debug] > > jk_shutdown_socket::jk_connect.c (722): About to shutdown socket 154 > > [Tue Sep 07 10:20:22.619 2010] [18806:46962404156768] [debug] > > jk_shutdown_socket::jk_connect.c (813): Shutdown socket 154 and read 0 > > lingering bytes in 2 sec. > > Hmmm, that's strange. Why is there a connection close between > wc_get_worker_for_name() and wc_maintain()? The first call doing > something with the connections would be wc_maintain(). And if it decides > to close sockets, it would log addiional statements. Nevertheless we can > see, that closing the socket really took 2 seconds (which is in fact a > builtin mod_jk timeout). That's interesting, because I thought this bug: https://issues.apache.org/bugzilla/show_bug.cgi?id=48169 Was addressed in 1.2.30 and this discusses a 2 second problem. > Are we sure, that those log lines are correct? This was generated by > mod_jk 1.2.30 without any code changes, right? I compiled it fresh from source without any change: [Tue Sep 07 10:07:56.683 2010] [17424:46962404156768] [info] init_jk::mod_jk.c (3189): mod_jk/1.2.30 initialized > Log level trace would show us more precisely, how we came to close the > socket, but it blows up log volume a lot. I'm using trace and greping for whatever you need to see. if you want to see more debug, tell me what keyword to grep (ideally, so I don't catch the hundreds of other lines of debug being generated by other worker threads). > > To confirm the settings: > > > > worker.basic.connection_pool_timeout=90 > > You don't usually want to set a pool_size. "1" would be appropriate when > using the prefork Apache MPM, but in that case mod_jk automatically sets > it to "1". I can remove it if you want - I only set it to 1 to ensure it was set (yes, I am using prefork). > > > worker.basic.connection_pool_size=1 > > worker.basic.socket_keepalive=1 > > I don't like "socket_timeout" ... Why? > > worker.basic.socket_timeout=90 > > but I would like socket_connect_timeout. I can change it. > The next two are possibly a bit short, because if the backend e.g. does > a Java Garbage Collection which miht take longer than 1 second, tose > timeouts will fire and take the node out of load balancing. > > > worker.basic.connect_timeout=1000 > > worker.basic.prepost_timeout=1000 > > You might want to add max_reply_timeouts, otherwise one single reply > timeout can take a node out of load balancing. > > > worker.basic.reply_timeout=9 > > # Stop recoery attempts when JBoss instances do not respond. > > worker.basic.retries=1 > > worker.basic.recovery_options=27 > Don't want sticky sessions? Nope. The app clusters. Although it's set for other workers. John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
TM, On Tuesday 07 September 2010 09:47:17 you wrote: > If you have enough resources, try to disable > connectionTimeout on AJP connector and see weather > the same will happen again. I've removed the connectionTimeout attribute and the problem persists - I can see the "lingering bytes in 2 sec" messages as before. John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
Rainer, Is this acceptable? I am using a tail and an egrep to match the various statements you wish to see. if it's missing anything, plesae let me know what to add to the grep. [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] find_match::jk_uri_worker_map.c (850): Attempting to map context URI '/*=lb-jboss51-integration' source 'JkMount' [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] find_match::jk_uri_worker_map.c (863): Found a wildchar match '/*=lb-jboss51-integration' [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] jk_handler::mod_jk.c (2462): Into handler jakarta-servlet worker=lb-jboss51-integration r->proxyreq=0 [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] wc_get_worker_for_name::jk_worker.c (116): found a worker lb-jboss51-integration [Tue Sep 07 10:20:20.618 2010] [18806:46962404156768] [debug] jk_shutdown_socket::jk_connect.c (722): About to shutdown socket 154 [Tue Sep 07 10:20:22.619 2010] [18806:46962404156768] [debug] jk_shutdown_socket::jk_connect.c (813): Shutdown socket 154 and read 0 lingering bytes in 2 sec. [Tue Sep 07 10:20:22.625 2010] [18806:46962404156768] [debug] wc_maintain::jk_worker.c (339): Maintaining worker lb-jboss51-integration [Tue Sep 07 10:20:22.628 2010] [18806:46962404156768] [debug] get_most_suitable_worker::jk_lb_worker.c (1001): found best worker jboss51-integration1 (jboss51-integration1) using method 'Request' [Tue Sep 07 10:20:22.628 2010] [18806:46962404156768] [debug] service::jk_lb_worker.c (1161): service worker=jboss51-integration1 route=jboss51-integration1 [Tue Sep 07 10:20:22.628 2010] [18806:46962404156768] [debug] ajp_service::jk_ajp_common.c (2376): processing jboss51-integration1 with 1 retries [Tue Sep 07 10:20:22.628 2010] [18806:46962404156768] [debug] ajp_send_request::jk_ajp_common.c (1579): (jboss51-integration1) all endpoints are disconnected. [Tue Sep 07 10:20:22.631 2010] [18806:46962404156768] [debug] ajp_send_request::jk_ajp_common.c (1639): (jboss51-integration1) request body to send 0 - request body to resend 0 [Tue Sep 07 10:20:22.637 2010] [18806:46962404156768] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1336): 00305F 2E 6A 62 6F 73 73 35 31 2D 69 6E 74 65 67 72 - _.jboss51-integr [Tue Sep 07 10:20:22.638 2010] [18806:46962404156768] [debug] ajp_unmarshal_response::jk_ajp_common.c (723): Header[0] [Set-Cookie] = [JSESSIONID=cM2w+W7dIfDERfQpzO6BDw__.jboss51-integration1; Path=/xx] [Tue Sep 07 10:20:22.640 2010] [18806:46962404156768] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1336): 01605F 2E 6A 62 6F 73 73 35 31 2D 69 6E 74 65 67 72 - _.jboss51-integr .. [Tue Sep 07 10:20:22.651 2010] [18806:46962404156768] [debug] ajp_reset_endpoint::jk_ajp_common.c (757): (jboss51-integration1) resetting endpoint with sd = 154 [Tue Sep 07 10:20:22.651 2010] [18806:46962404156768] [debug] ajp_done::jk_ajp_common.c (3010): recycling connection pool slot=0 for worker jboss51-integration1 [Tue Sep 07 10:20:22.651 2010] [18806:46962404156768] [debug] jk_handler::mod_jk.c (2602): Service finished with status=200 for worker=lb-jboss51-integration To confirm the settings: worker.basic.connection_pool_timeout=90 worker.basic.connection_pool_size=1 worker.basic.socket_keepalive=1 worker.basic.socket_timeout=90 worker.basic.connect_timeout=1000 worker.basic.prepost_timeout=1000 worker.basic.reply_timeout=9 # Stop recoery attempts when JBoss instances do not respond. worker.basic.retries=1 worker.basic.recovery_options=27 worker.jboss51-integration1.reference=worker.basic worker.jboss51-integration1.host=xx worker.jboss51-integration1.port=13802 worker.jboss51-integration1.type=ajp13 worker.jboss51-integration1.lbfactor=1 worker.jboss51-integration2.reference=worker.basic worker.jboss51-integration2.host=yy worker.jboss51-integration2.port=13802 worker.jboss51-integration2.type=ajp13 worker.jboss51-integration2.lbfactor=1 worker.lb-jboss51-integration.type=lb worker.lb-jboss51-integration.balance_workers=jboss51-integration1,jboss51-integration2 worker.lb-jboss51-integration.sticky_session=0 worker.list=lb-jboss51-integration Also, I've tried setting: worker.basic.lock=0 but that does not solve the problem. Thanks John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Tuesday 07 September 2010 08:59:27 you wrote: > It means that socket shutdown failed. > Do you have firewall between mod_jk and JBoss or some non-standard > network driver (e.g running under some VM)? We are using VMs but there should be no firewall. I should probably re-iterate that the problem is occassional. > This can cause network FIN packet not getting transmitted over it > since we issued socket shutdown, but the other side didn't react on > that message. > > What kind of connector is on the JBoss side. Java or APR? APR? It's a standard JBoss 5.1.2 installation. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Tuesday 07 September 2010 06:28:33 you wrote: > On 09/06/2010 11:59 PM, John Baker wrote: > > What's jk_maintain? > > > > Function that maintains the workers > (closes excess connections inactive for a long time) > > Anyhow, like Rainer said, if that's the case > you should have something like this in the log (DEBUG) > > ... Shutdown socket 123 and read 123 lingering bytes in 2 sec. Is this what we're looking for? There's a lot of it in the logs during the period in which I switched on 'trace'. [Mon Sep 06 15:05:18.128 2010] [24061:1404667232] [debug] jk_shutdown_socket::jk_connect.c (732): Shutdown socket 158 and rea 0 lingering bytes If so, what does this mean and how do I solve the problem? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Monday 06 September 2010 22:57:21 you wrote: > I didn't look at the code now, but the 2 seconds remind me of the > connection draining during socket shutdown, which could be related to > jk_maintain? What's jk_maintain? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 second delays in mod_jk while "maintaining workers"
On Monday 06 September 2010 18:56:20 you wrote: > On 09/06/2010 04:16 PM, John Baker wrote: > > > > I've set the Jk logging to trace and you can see the debug statements and > > the 2s delays: > > > > Do you use NFS share by any chance to store the > mod_jk log directory data? Nope. All log files are written to local disc, and they are very quick discs.. > What happens if you set > worker.lb-jboss51-integration.lock=O I'll get back to you and let you know. By the way, there's a lot of activity on these Apache servers - do you suspect a file lock timeout? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
2 second delays in mod_jk while "maintaining workers"
Hello, I've discovered what appears to be a bug in mod_jk 1.2.27 and have also tried 1.2.30 without success. I'm using Apache 2.2.3 (on Redhat EL 5.4). The problem occurs after previous successful activity and causes a delay in what looks like socket handling. I noticed bug was resolved in 1.2.30 with respect to sockets but this does not appear to have solved the problem. I've set the Jk logging to trace and you can see the debug statements and the 2s delays: [Mon Sep 06 15:05:12.121 2010] [24061:1404667232] [debug] wc_get_worker_for_name::jk_worker.c (116): found a worker lb-jboss51-integration [Mon Sep 06 15:05:14.128 2010] [24061:1404667232] [debug] wc_maintain::jk_worker.c (339): Maintaining worker lb-jboss51-integration [Mon Sep 06 15:05:14.241 2010] [23945:1404667232] [debug] wc_maintain::jk_worker.c (339): Maintaining worker lb-jboss51-integration [Mon Sep 06 15:05:16.644 2010] [23944:1404667232] [debug] wc_maintain::jk_worker.c (339): Maintaining worker lb-jboss51-integration [Mon Sep 06 15:05:18.131 2010] [24061:1404667232] [debug] init_ws_service::mod_jk.c (977): Service protocol=HTTP/1.1 method=GET ssl=false host=(null) addr=10.117.112.1 name=xxx port=80 auth=(null) user=(null) laddr=10.117.112.1 raddr=10.117.112.1 uri=/index.jsp [Mon Sep 06 15:05:18.131 2010] [24061:1404667232] [debug] get_most_suitable_worker::jk_lb_worker.c (997): found best worker jboss51-integration2 (jboss51-integration2) using method 'Request' Here's the workers.properties config: worker.basic.connection_pool_timeout=90 worker.basic.connection_pool_size=1 worker.basic.socket_keepalive=1 worker.basic.socket_timeout=90 worker.basic.connect_timeout=1000 worker.basic.prepost_timeout=1000 worker.basic.reply_timeout=9 worker.basic.retries=1 worker.basic.recovery_options=27 worker.jboss51-integration1.reference=worker.basic worker.jboss51-integration1.host=xx worker.jboss51-integration1.port=13802 worker.jboss51-integration1.type=ajp13 worker.jboss51-integration1.lbfactor=1 worker.jboss51-integration2.reference=worker.basic worker.jboss51-integration2.host=yy worker.jboss51-integration2.port=13802 worker.jboss51-integration2.type=ajp13 worker.jboss51-integration2.lbfactor=1 worker.lb-jboss51-integration.type=lb worker.lb-jboss51-integration.balance_workers=jboss51-integration1,jboss51-integration2 worker.lb-jboss51-integration.sticky_session=0 worker.list=lb-jboss51-integration I've tried replacing connect_timeout, prepost_timeout and reply_timeout with ping_timeout, but this does not resolve the matter. I've tried commenting out various socket/connection settings, but have come to no firm conclusion to what may be at fault. The JBoss (5.1, but also a problem with 4.2.3) server.xml file has connectionTimeout="9" set on the AJP connetor. Can anyone offer some advice? Or if I need to provide more information, please detail what is required. Many thanks, John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
