mod_proxy race condition bug #37770
Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 I think this bug is a serious issue for our application so I'm trying to see if anyone has figured out a good way to actually fix this problem. The patch mentioned in #82 is not a fix for the actual problem caused by the race condition. Sounds to me like this race condition can't really be solved though...does anyone know what other proxy software does in these cases with keepalive connections? Cheers, Adam
Re: mod_proxy race condition bug #37770
Reading over the bug I'm still curious about the actual race condition aspect of it... Even doing an 'OPTIONS *' right before sending the request does not ensure that the race condition goes away because there is still that period of time where the backend could go away right after that (in fact, it also complicates things since we need to worry about the keepalive aspects)... The best solution is to simply trap that error and have Apache do the right thing, which is what r645813 does. Ruediger and I tested his patch @ ApacheCon and it works pretty well. On May 12, 2008, at 1:52 PM, Adam Woodworth wrote: Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 I think this bug is a serious issue for our application so I'm trying to see if anyone has figured out a good way to actually fix this problem. The patch mentioned in #82 is not a fix for the actual problem caused by the race condition. Sounds to me like this race condition can't really be solved though...does anyone know what other proxy software does in these cases with keepalive connections? Cheers, Adam
Re: mod_proxy race condition bug #37770
On Mon, 12 May 2008 13:52:18 -0400 Adam Woodworth [EMAIL PROTECTED] wrote: Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 If this is indeed a race condition biting you, it's possible you could fix it by setting a sufficiently low ttl value that the backend's ttl will be guaranteed to outlive it - depending of course on backend behaviour. So if the backend has a 15 second ttl, set 10 secs in the proxy, and the remaining 5 secs provide a good buffer against a race condition. Except that ttl doesn't work exactly as documented. There's a similar issue with ttl and DBD database connections. I've just posted on the subject to [EMAIL PROTECTED] (apr_reslist semantics). This could enable you to configure apache to avoid this problem without sacrificing backend keepalives altogether. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: mod_proxy race condition bug #37770
I haven't looked at the code in mod_proxy to see how it handles the Keep-Alive header returned by the backend server, but what I'm seeing in this tcpdump I have that shows the proxy error happening is that the backend webserver is IIS, and it is not sending any Connection or Keep-Alive headers back to mod_proxy, even though mod_proxy sent a Connection: Keep-Alive header. But, this is HTTP 1.1 -- is the server required to send back a Connection and Keep-Alive header? What I'm wondering is, if the server doesn't send back a Keep-Alive header that specifies the timeout and max requests (e.g., Keep-Alive: timeout=15, max=500), then does mod_proxy just default to an infinite (or very large) timeout for that connection? And somehow it's not noticing that the server closed the connection? What I see is that in a nearly 700 second long tcp dump, there are a few mod_proxy requests made to the backend w/o any SYN/SYN-ACK/ACK TCP handshaking happening before the GET packet, which means that Apache must think that the socket is still open -- i.e., it used the socket for a connection a bunch of minutes ago, and still thinks it's alive. But I see this happening 600 seconds into the dump, without any other activity on the same socket prior to that in the dump, which leads me to think that Apache is just keeping the socket around for quite a long time. But it surprises me that IIS hasn't closed the socket by then, so that by the time Apache tried to use it it would notice it was closed...hmmm, very strange. Adam On Mon, May 12, 2008 at 3:31 PM, Nick Kew [EMAIL PROTECTED] wrote: On Mon, 12 May 2008 13:52:18 -0400 Adam Woodworth [EMAIL PROTECTED] wrote: Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 If this is indeed a race condition biting you, it's possible you could fix it by setting a sufficiently low ttl value that the backend's ttl will be guaranteed to outlive it - depending of course on backend behaviour. So if the backend has a 15 second ttl, set 10 secs in the proxy, and the remaining 5 secs provide a good buffer against a race condition. Except that ttl doesn't work exactly as documented. There's a similar issue with ttl and DBD database connections. I've just posted on the subject to [EMAIL PROTECTED] (apr_reslist semantics). This could enable you to configure apache to avoid this problem without sacrificing backend keepalives altogether. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: svn commit: r655594 - /httpd/httpd/branches/2.2.x/STATUS
On 05/12/2008 08:39 PM, [EMAIL PROTECTED] wrote: Author: jim Date: Mon May 12 11:39:34 2008 New Revision: 655594 URL: http://svn.apache.org/viewvc?rev=655594view=rev Log: backport proposal: http://svn.apache.org/viewvc?view=revrevision=645813 Modified: httpd/httpd/branches/2.2.x/STATUS Modified: httpd/httpd/branches/2.2.x/STATUS URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?rev=655594r1=655593r2=655594view=diff == --- httpd/httpd/branches/2.2.x/STATUS (original) +++ httpd/httpd/branches/2.2.x/STATUS Mon May 12 11:39:34 2008 @@ -187,6 +187,16 @@ bump. Not sure what style and typo changes are required. jim: Latest rev of patch should address stated concerns :) + * mod_proxy: In the case that we fail to read the response line + from the backend and if we are a reverse proxy request, shutdown + the connection WITHOUT ANY response to trigger a retry by the client + if allowed (as for idempotent requests). + Trunk version of patch: + http://svn.apache.org/viewvc?view=revrevision=645813 + Backport version for 2.2.x of patch: + Trunk version works + +1: I assume you are +1 as well :-)? Regards Rüdiger
