mod_proxy fails to send FIN response when a FIN is received from a backend (bug 50807)
Hi, Has anyone been able to look into bug #50807? It is a serious issue for us that has shown up a number of times, some causing serious problems with firewalls and load balancers preventing traffic from getting through at all. Thanks, Adam
Re: mod_proxy race condition bug #37770
Sounds good, glad I was able to help. I'll keep an eye on this in a future httpd release. On Sat, Sep 20, 2008 at 6:42 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 09/20/2008 12:21 PM, Nick Kew wrote: On 19 Sep 2008, at 23:08, Ruediger Pluem wrote: On 09/19/2008 11:24 PM, Adam Woodworth wrote: The problem with this is that it will also do this for connections to the backend that timed out. For our application, we want actual timeouts to be caught as normal and kicked back through to the client as a 50x error. Hm. Good point. I have to think about it, because a timeout error can justify a repeated request (like when a firewall between frontend and backend cut the connection and now the packets are simply dropped by the firewall and a fresh one would fix this). I think Adam is right. The justification for the RFC-breaking connection drop is IIRC that it's merely propagating the same from the backend to the client. That doesn't apply in the case of a timeout. Yes, I came to the same conclusion in the meantime. The firewall drop can be prevented by setting keepalive to on. Furthermore the typical keepalive timeouts for http connections are way below the typical firewall timeouts. So if we get back APR_TIMEUP we can safely assume that the backend did not respond in a timely manner. Looking at his patch I think it looks basically fine. Hope to find some time for a closer look and a commit later on. Regards Rüdiger
Re: mod_proxy race condition bug #37770
From a usage standpoint I have no real preference other than I need to be able to configure it from httpd.conf on a per-host basis. :) What is a worker specific property? I would like to fold the approach of the patch into trunk to reduce PR37770 to become (hopefully) an RTFM bug. The approach of the patch above in conjunction with the reslist TTL fix in upcoming APR-UTIL 1.3.x should get us there. Any opinions about how to make this configurable (environment variable or via a worker specific property)? Regards Rüdiger
Re: mod_proxy race condition bug #37770
Can you please test the attached patch in addition to my other patches, but without removing mod_deflate from the chain as you did in your application? Thanks. This patch works for us, we don't have to remove the deflate filter ourselves anymore. Thanks! Will this also make it into 2.2.9? Cheers, Adam
Re: mod_proxy race condition bug #37770
I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
And FWIW, when the EOC bucket is sent down, our mod_perl filter sees an ERROR bucket, i.e. the bucket type is ERROR not EOC, if that matters. On Fri, May 23, 2008 at 1:19 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
Err, sorry, we do see the EOC bucket, it's just after the ERROR and EOS buckets (duh). :) On Fri, May 23, 2008 at 1:31 PM, Adam Woodworth [EMAIL PROTECTED] wrote: And FWIW, when the EOC bucket is sent down, our mod_perl filter sees an ERROR bucket, i.e. the bucket type is ERROR not EOC, if that matters. On Fri, May 23, 2008 at 1:19 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I think that the deflate filter might need to be fixed in order to work with this patch, I think the EOC bucket is causing the deflate filter to cause apache to return a 200 OK to the client with a blank page. I had to make our mod_perl application remove the deflate filter from the chain when we got an EOC in order to make this work. So perhaps the deflate filter needs to handle this itself. And perhaps the same problem will affect many other filters. On Thu, May 22, 2008 at 4:57 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
Thanks Ruediger. Keep us posted. Is this mod_proxy patch going to get into 2.2.9, or wait until another 2.2.x release when the deflate issue can be resolved? Cheers, Adam On Fri, May 23, 2008 at 5:51 PM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/23/2008 07:55 PM, Adam Woodworth wrote: Err, sorry, we do see the EOC bucket, it's just after the ERROR and EOS buckets (duh). :) This is what I expect. I have checked and I think I know what's going wrong, but I have no solution yet how to fix it. The solution I am looking for should not involve a change to the mod_deflate filter or the need to remove it from the chain. Regards Rüdiger
Re: mod_proxy race condition bug #37770
Ok my mistake, the setup I was using turned out to be the problem. I'm now able to see the patch working. Sorry about that. Thanks again Ruediger. Now I just have to modify our mod_perl application to behave correctly when mod_proxy sends EOC. Adam On Thu, May 22, 2008 at 4:28 AM, Ruediger Pluem [EMAIL PROTECTED] wrote: On 05/21/2008 10:34 PM, Adam Woodworth wrote: I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means No, it is not always 0 at this point. It has a different value once the connection to the client is not used for the first time. that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere? Yes, with the test code I posted here on list. Regards Rüdiger
Re: mod_proxy race condition bug #37770
I think there is a problem with r657443 (and thus I assume r645813): I applied the r657443 patch to my copy of the 2.2.8 official release source and it doesn't work right. The problem is that the change to mod_proxy_http.c checks c-keepalives for a value, but c-keepalives is filled out by ap_http_header_filter(), which isn't called until later. So, c-keepalives is always 0 at this point. Which also means that the number of keepalives message in the ap_log_rerror() msg isn't going to mean anything, but at this point it never reaches it. Also, I patched the patch to manually set c-keepalives to 1 for testing so that I could test out this r657443 patch. However, what happened was that Apache would kick back an error page to the client, instead of just closing the connection. So, no improvement. Has this patch been tested and verified anywhere?
svn.apache.org down?
I very regularly have problems accessing svn.apache.org. For example, right now a traceroute gets to corv-car1-gw.nero.net and then can't get any further. I can't access svn.apache.org at all right now. Which means I can't do some work right now. :) Does anyone know of any problems with svn.apache.org or its ISP?
Re: svn.apache.org down?
Hi, While I'm waiting for my subscription to infra to be activated... There are no errors -- I simply cannot connect to svn.apache.org. I am in Boston, MA, in the USA. The traceroute never finishes...the browser never connects...etc...it's like the net link to svn.apache.org is just not working. This page seems to show that eris.apache.org (same IP as svn.apache.org) is down: http://monitoring.apache.org/status/ Adam On Mon, May 19, 2008 at 11:43 AM, Tony Stevenson [EMAIL PROTECTED] wrote: Adam Woodworth wrote: I very regularly have problems accessing svn.apache.org. For example, right now a traceroute gets to corv-car1-gw.nero.net and then can't get any further. I can't access svn.apache.org at all right now. Which means I can't do some work right now. :) Does anyone know of any problems with svn.apache.org or its ISP? There is no known issue with svn.apache.org at the moment Adam. However there have been a couple of difficulties with the network interface these past few days. If you are in the EU, you can try accessing svn.eu.apache.org Can you be more specific as to what exactly is not working? Error messages would be useful. We are trying to stabilise svn.apache.org as fast as we can, but this can be trecherous when it is live, and in production. The EU svn mirror should be fully online shortly, however with the issues with svn.apache.org, yo may see a slight delay in the changes being sync'd to the master. Adam, fwiw, emailing [EMAIL PROTECTED] may yield better replies. Even though most of us in infra read [EMAIL PROTECTED] --Tony
Re: mod_proxy race condition bug #37770
I tried using the SetEnv proxy-nokeepalive 1 option in httpd.conf, and it cleared up the proxy errors that I was having with an IIS backend server, and it may have decreased the proxy errors with Apache backend servers as well. I tried again without the proxy-nokeepalive option, and the proxy errors increased. Put it back, they decreased dramatically again. Turns out that mod_proxy always sends Connection: Keep-Alive, but with proxy-nokeepalive 1, mod_proxy will always send Connection: Close. The backends in my case are all responding with Connection: Close, so the backends aren't even allowing Keep-Alives, regardless of whether mod_proxy sends Connection: Close or Connection: Keep-Alive. So you have either set force-proxy-request-1.0 and thus are doing HTTP/1.0 requests to your backend or your backend has keepalives disabled. No, like I said above, I'm using proxy-nokeepalive to force mod_proxy to send Connection: Close.
Re: mod_proxy race condition bug #37770
And FWIW the backends also seem to have keepalives disabled. On Mon, May 19, 2008 at 12:36 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I tried using the SetEnv proxy-nokeepalive 1 option in httpd.conf, and it cleared up the proxy errors that I was having with an IIS backend server, and it may have decreased the proxy errors with Apache backend servers as well. I tried again without the proxy-nokeepalive option, and the proxy errors increased. Put it back, they decreased dramatically again. Turns out that mod_proxy always sends Connection: Keep-Alive, but with proxy-nokeepalive 1, mod_proxy will always send Connection: Close. The backends in my case are all responding with Connection: Close, so the backends aren't even allowing Keep-Alives, regardless of whether mod_proxy sends Connection: Close or Connection: Keep-Alive. So you have either set force-proxy-request-1.0 and thus are doing HTTP/1.0 requests to your backend or your backend has keepalives disabled. No, like I said above, I'm using proxy-nokeepalive to force mod_proxy to send Connection: Close.
Re: mod_proxy race condition bug #37770
So let's consider the case of a web browser that uses keepalives. If
the web browser has a keepalive connection, and the connection closes
behind it's back so that the next time the browser tries to use the
connection it fails (like this problem we're having with mod_proxy),
what should the web browser do?
I bring this up because while I certainly never see Firefox regularly
fail to load pages, I'm seeing mod_proxy fail with some regularity,
perhaps only a fraction of a percent, but still... And you would
think you'd see the same problem with any sort of browser/proxy.
Has anyone seen the socket code for Firefox to see if they're doing
something more clever to prevent dead connections from being used?
On Mon, May 19, 2008 at 3:18 PM, Ruediger Pluem [EMAIL PROTECTED] wrote:
On 05/19/2008 06:09 PM, Adam Woodworth wrote:
Index: modules/proxy/proxy_util.c
+/* Close a possible existing socket if we are told to do so */
+if (conn-close) {
+socket_cleanup(conn);
+conn-close = 0;
+}
Does this mean that sockets that should have been closed (via the
close flag) weren't getting closed correctly before? I.e. sockets in
the pool were still open as far as mod_proxy was concerned even
though we set the close flag?
No, it does not mean this. This patch is needed to correctly act, when the
patch
below kicks in. So far conn-close was *always* 0 at this point of code, but
with the patch below it might not, as we might decide *before* reusing a
connection
that we better should not.
Why is the patch below necessary? The comments indicate that if we
were to us an existing keepalive connection to the backend server, but
it failed, then if the client was NOT keepalive then we couldn't send
the failure to the client...but why? If we fail to connect to the
This is because the client does not expect this kind of failure in this
situation and
acts wrong. If it is the initial connection e.g. of a seamonkey browser it
does not
try to resent the request but simply displays an blank page which is bad for
the user.,
backend, whether on a keepalive or non-keepalive connection, why does
it matter if the client is keepalive or not in order to send an error?
We do not send an error, but we are closing the connection an expect to the
client to handle
this correctly which it does only if it used a kept alive connection.
Regards
Rüdiger
Re: mod_proxy race condition bug #37770
#include sys/types.h
#include sys/socket.h
#include netinet/ip.h
#include string.h
#include unistd.h
#include stdio.h
int main(int argc, char *argv[])
{
int listen_fd;
int fd;
struct sockaddr_storage addr;
socklen_t addrlen = sizeof(addr);
struct sockaddr_in addr_in;
struct linger l = {1, 0};
int val = 1;
char buffer[1024];
char response[] = HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n;
listen_fd = socket(PF_INET, SOCK_STREAM, 0);
memset(addr_in, 0, sizeof(struct sockaddr_in));
addr_in.sin_family = AF_INET;
addr_in.sin_port = ntohs(8080);
setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, val, sizeof val);
bind(listen_fd, addr_in, sizeof(addr_in));
listen(listen_fd, 50);
fd = accept(listen_fd, addr, addrlen);
read(fd, buffer, sizeof(buffer));
write(fd, (void *)response, strlen(response));
printf(%s\n, response);
read(fd, buffer, sizeof(buffer));
setsockopt(fd, SOL_SOCKET, SO_LINGER, l, sizeof l);
close(fd); /* sends TCP RST rather than FIN */
return 0;
}
It just closes down the connection with a RST on the second request on the
same connection. You will notice in the network sniffer that the browser
opens
a new TCP connection and resends the request.
Hah, you beat me to it. I was going to do the same thing. Thanks Rudiger!
Adam
Re: mod_proxy race condition bug #37770
Can anyone explain why turning on proxy-nokeepalive would cause proxy errors to an IIS server to stop happening? We would see proxy errors to IIS backends regularly with proxy-nokeepalive OFF (mod_proxy sends Connection: Keep-Alive). When we use proxy-nokeepalive ON (set to 1, mod_proxy sends Connetion: Close) the proxy errors between mod_proxy and the backend IIS servers stop. What would cause this?
Re: mod_proxy race condition bug #37770
Forgot an important detail: In both cases, the backend always responds with Connection: Close (it's setup to do that). On Mon, May 19, 2008 at 4:51 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Can anyone explain why turning on proxy-nokeepalive would cause proxy errors to an IIS server to stop happening? We would see proxy errors to IIS backends regularly with proxy-nokeepalive OFF (mod_proxy sends Connection: Keep-Alive). When we use proxy-nokeepalive ON (set to 1, mod_proxy sends Connetion: Close) the proxy errors between mod_proxy and the backend IIS servers stop. What would cause this?
Re: mod_proxy race condition bug #37770
So I have some more information about this, but this time related to having keepalives OFF in mod_proxy. I tried using the SetEnv proxy-nokeepalive 1 option in httpd.conf, and it cleared up the proxy errors that I was having with an IIS backend server, and it may have decreased the proxy errors with Apache backend servers as well. I tried again without the proxy-nokeepalive option, and the proxy errors increased. Put it back, they decreased dramatically again. Turns out that mod_proxy always sends Connection: Keep-Alive, but with proxy-nokeepalive 1, mod_proxy will always send Connection: Close. The backends in my case are all responding with Connection: Close, so the backends aren't even allowing Keep-Alives, regardless of whether mod_proxy sends Connection: Close or Connection: Keep-Alive. My theory is that mod_proxy is incorrectly using a socket pool even when the server responds with Connection: Close or when mod_proxy itself sends Connection: Close. Most of the time it probably realizes that the sockets in the pool are actually closed, but sometimes not and it tries to use a dead socket. What should be happening is that mod_proxy needs to NOT use a socket pool if Connection: Close is in use. However, I have not looked very deeply into the code to see if my theory is accurate. Could someone more familiar with the mod_proxy code please fill me in? Does mod_proxy always use a connection pool? Thanks, Adam On Mon, May 12, 2008 at 4:15 PM, Adam Woodworth [EMAIL PROTECTED] wrote: I haven't looked at the code in mod_proxy to see how it handles the Keep-Alive header returned by the backend server, but what I'm seeing in this tcpdump I have that shows the proxy error happening is that the backend webserver is IIS, and it is not sending any Connection or Keep-Alive headers back to mod_proxy, even though mod_proxy sent a Connection: Keep-Alive header. But, this is HTTP 1.1 -- is the server required to send back a Connection and Keep-Alive header? What I'm wondering is, if the server doesn't send back a Keep-Alive header that specifies the timeout and max requests (e.g., Keep-Alive: timeout=15, max=500), then does mod_proxy just default to an infinite (or very large) timeout for that connection? And somehow it's not noticing that the server closed the connection? What I see is that in a nearly 700 second long tcp dump, there are a few mod_proxy requests made to the backend w/o any SYN/SYN-ACK/ACK TCP handshaking happening before the GET packet, which means that Apache must think that the socket is still open -- i.e., it used the socket for a connection a bunch of minutes ago, and still thinks it's alive. But I see this happening 600 seconds into the dump, without any other activity on the same socket prior to that in the dump, which leads me to think that Apache is just keeping the socket around for quite a long time. But it surprises me that IIS hasn't closed the socket by then, so that by the time Apache tried to use it it would notice it was closed...hmmm, very strange. Adam On Mon, May 12, 2008 at 3:31 PM, Nick Kew [EMAIL PROTECTED] wrote: On Mon, 12 May 2008 13:52:18 -0400 Adam Woodworth [EMAIL PROTECTED] wrote: Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 If this is indeed a race condition biting you, it's possible you could fix it by setting a sufficiently low ttl value that the backend's ttl will be guaranteed to outlive it - depending of course on backend behaviour. So if the backend has a 15 second ttl, set 10 secs in the proxy, and the remaining 5 secs provide a good buffer against a race condition. Except that ttl doesn't work exactly as documented. There's a similar issue with ttl and DBD database connections. I've just posted on the subject to [EMAIL PROTECTED] (apr_reslist semantics). This could enable you to configure apache to avoid this problem without sacrificing backend keepalives altogether. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
mod_proxy race condition bug #37770
Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 I think this bug is a serious issue for our application so I'm trying to see if anyone has figured out a good way to actually fix this problem. The patch mentioned in #82 is not a fix for the actual problem caused by the race condition. Sounds to me like this race condition can't really be solved though...does anyone know what other proxy software does in these cases with keepalive connections? Cheers, Adam
Re: mod_proxy race condition bug #37770
I haven't looked at the code in mod_proxy to see how it handles the Keep-Alive header returned by the backend server, but what I'm seeing in this tcpdump I have that shows the proxy error happening is that the backend webserver is IIS, and it is not sending any Connection or Keep-Alive headers back to mod_proxy, even though mod_proxy sent a Connection: Keep-Alive header. But, this is HTTP 1.1 -- is the server required to send back a Connection and Keep-Alive header? What I'm wondering is, if the server doesn't send back a Keep-Alive header that specifies the timeout and max requests (e.g., Keep-Alive: timeout=15, max=500), then does mod_proxy just default to an infinite (or very large) timeout for that connection? And somehow it's not noticing that the server closed the connection? What I see is that in a nearly 700 second long tcp dump, there are a few mod_proxy requests made to the backend w/o any SYN/SYN-ACK/ACK TCP handshaking happening before the GET packet, which means that Apache must think that the socket is still open -- i.e., it used the socket for a connection a bunch of minutes ago, and still thinks it's alive. But I see this happening 600 seconds into the dump, without any other activity on the same socket prior to that in the dump, which leads me to think that Apache is just keeping the socket around for quite a long time. But it surprises me that IIS hasn't closed the socket by then, so that by the time Apache tried to use it it would notice it was closed...hmmm, very strange. Adam On Mon, May 12, 2008 at 3:31 PM, Nick Kew [EMAIL PROTECTED] wrote: On Mon, 12 May 2008 13:52:18 -0400 Adam Woodworth [EMAIL PROTECTED] wrote: Hi, I was wondering if anyone might have some more information on bug #37770. I've added a comment there recently, at the end of the bug report, #83. https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 If this is indeed a race condition biting you, it's possible you could fix it by setting a sufficiently low ttl value that the backend's ttl will be guaranteed to outlive it - depending of course on backend behaviour. So if the backend has a 15 second ttl, set 10 secs in the proxy, and the remaining 5 secs provide a good buffer against a race condition. Except that ttl doesn't work exactly as documented. There's a similar issue with ttl and DBD database connections. I've just posted on the subject to [EMAIL PROTECTED] (apr_reslist semantics). This could enable you to configure apache to avoid this problem without sacrificing backend keepalives altogether. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: Issues with mod_proxy_http, keep-alive, and SSL
1st question still up for grabs, but I found the answer to my 2nd question for those interested: Bug 43472 (http://issues.apache.org/bugzilla/show_bug.cgi?id=43472) reported this problem and the fix for it is in 2.2.7. I just tried the 2.2.7 tag from svn and it solved the issue for me. On Jan 8, 2008 7:10 PM, Adam Woodworth [EMAIL PROTECTED] wrote: Hi, I have a couple issues with mod_proxy: 1) HTTP Keep-Alive on an SSL Connection: In the source for Apache 2.2.6, around line 1704 of modules/proxy/mod_proxy_http.c there is this code that causes HTTPS connections to not use Keep-Alive's: backend-is_ssl = is_ssl; /* * TODO: Currently we cannot handle persistent SSL backend connections, * because we recreate backend-connection for each request and thus * try to initialize an already existing SSL connection. This does * not work. */ if (is_ssl) backend-close_on_recycle = 1; Looking in the same function in the latest Apache 2.0.x source, in file modules/proxy/proxy_http.c around line 1081, the check for setting close_on_recycle doesn't exist (there is no close_on_recycle in the source). I'm not intimately familiar with the differences between mod_proxy in 2.0.x and 2.2.x, but does this mean that the bug described (and prevented) above in the 2.2.6 code (Keep-Alive persistent connections with SSL in mod_proxy won't work with the current 2.2.6 design, and are thus prevented) is still an issue (was never fixed) in 2.0.x? 2) Using Apache 2.2.6, mod_proxy opens a new connection for each request being proxied: This is the same exact problem that is described in this bug from 2006 (except our backend isn't JBoss, it's just another Apache web server): http://issues.apache.org/bugzilla/show_bug.cgi?id=38602 To summarize, if a client (A) requests a keep-alive connection to our Apache proxy (B) (running httpd 2.2.6), mod_proxy still opens a new connection to the back-end (C) for each request coming from the client, even though the client has keep-alive connections open. Simple scenario: I connect from A to B over a single connection (one socket) and use HTTP/1.1 keep-alive requests to GET 4 URLs. For each request that A sends B over that single connection, B creates a new request to C. I would expect only one connection to be opened from B to C, to mirror what A is requesting of B. This problem, described in bug 38602, was solved for that individual with the patch made at that time that is in the code for 2.2.6. So it's strange that I'm seeing the same issue now. Anyone have any insight here? Thanks! Adam
Issues with mod_proxy_http, keep-alive, and SSL
Hi, I have a couple issues with mod_proxy: 1) HTTP Keep-Alive on an SSL Connection: In the source for Apache 2.2.6, around line 1704 of modules/proxy/mod_proxy_http.c there is this code that causes HTTPS connections to not use Keep-Alive's: backend-is_ssl = is_ssl; /* * TODO: Currently we cannot handle persistent SSL backend connections, * because we recreate backend-connection for each request and thus * try to initialize an already existing SSL connection. This does * not work. */ if (is_ssl) backend-close_on_recycle = 1; Looking in the same function in the latest Apache 2.0.x source, in file modules/proxy/proxy_http.c around line 1081, the check for setting close_on_recycle doesn't exist (there is no close_on_recycle in the source). I'm not intimately familiar with the differences between mod_proxy in 2.0.x and 2.2.x, but does this mean that the bug described (and prevented) above in the 2.2.6 code (Keep-Alive persistent connections with SSL in mod_proxy won't work with the current 2.2.6 design, and are thus prevented) is still an issue (was never fixed) in 2.0.x? 2) Using Apache 2.2.6, mod_proxy opens a new connection for each request being proxied: This is the same exact problem that is described in this bug from 2006 (except our backend isn't JBoss, it's just another Apache web server): http://issues.apache.org/bugzilla/show_bug.cgi?id=38602 To summarize, if a client (A) requests a keep-alive connection to our Apache proxy (B) (running httpd 2.2.6), mod_proxy still opens a new connection to the back-end (C) for each request coming from the client, even though the client has keep-alive connections open. Simple scenario: I connect from A to B over a single connection (one socket) and use HTTP/1.1 keep-alive requests to GET 4 URLs. For each request that A sends B over that single connection, B creates a new request to C. I would expect only one connection to be opened from B to C, to mirror what A is requesting of B. This problem, described in bug 38602, was solved for that individual with the patch made at that time that is in the code for 2.2.6. So it's strange that I'm seeing the same issue now. Anyone have any insight here? Thanks! Adam
