Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On 02/20/2006 02:51 PM, Joe Orton wrote: That's not what I meant. This is a bit tricky. Here's a restatement of the problem, just to make sure we're talking about the same thing: If a client reuses a connection to an HTTP server which has been held open persistently, there is a risk that the connection may be closed either beforehand or simultaneously to when the client tries to reuse it. So, it's quite normal to start sending a POST request with a body, and then half way through sending the request body, you get an TCP RST or a FIN. HTTP/1.1 clients have logic to open a new connection and resend the request if that happens. If the client in this case is the proxy, then the proxy may not be able to resend the whole request body if it is streaming it through and may already have discarded the first half by the time it sees the RST/FIN/. If the proxy has the correct logic then this can be handled correctly. In handling a request which includes a body (and the body will not/cannot be cached entirely in memory): There is an interesting bug regarding this topic: http://issues.apache.org/bugzilla/show_bug.cgi?id=38763 It seems that we do not have to worry about resending POSTs at all because of 8.1.4 of RFC2616 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html) This means that clients, servers, and proxies MUST be able to recover from asynchronous close events. Client software SHOULD reopen the transport connection and retransmit the aborted sequence of requests without user interaction so long as the request sequence is idempotent (see section 9.1.2). Non-idempotent methods or sequences MUST NOT be automatically retried, although user agents MAY offer a human operator the choice of retrying the request(s). Confirmation by user-agent software with semantic understanding of the application MAY substitute for user confirmation. The automatic retry SHOULD NOT be repeated if the second sequence of requests fails. POST does not seem to be an idempotent method to me. Even retransmitting GETs with parameters in the URL seems dangerous to me. a) if the connection to the client has been kept open persistently (i.e. r-connection-keepalives 1), then it is safe to forward the request on a connection to the backend which has itself been held open persistently. If the backend connection is closed whilst sending the request/body, then the connection to the client should also immediately be closed without sending a response. b) if the connection to the client has *not* been kept open persistently, then it is only safe to forward a request which includes a request body on a newly opened connection to the backend. (since such a connection is not vulnerable to a persistent connection timeout) Your proposals still make sense as we let the real client decide to do the dirty decision work whether to retransmit or not (for comments to them see http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/[EMAIL PROTECTED]). Regards Rüdiger
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On Wed, Feb 15, 2006 at 04:44:43PM -, Jim Jagielski wrote: Author: jim Date: Wed Feb 15 08:44:42 2006 New Revision: 378032 URL: http://svn.apache.org/viewcvs?rev=378032view=rev Log: *) mod_proxy: Fix KeepAlives not being allowed and set to backend servers. PR38602. [Ruediger Pluem, Jim Jagielski] Also, document previous patch: *) Correctly initialize mod_proxy workers, which use a combination of local and shared datasets. Adjust logging to better trace usage. PR38403. [Jim Jagielski] This one seems to have broken the proxy tests again, failed tests are: t/ssl/proxy.t 172 58 33.72% 3 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 115- 116 118-120 122 124 126 128 130 132 134 136 139-140 143-144 147-148 151-152 155- 156 159-160 163-164 167-168 171-172 if I svn up -r378031 modules/proxy this goes away. This looks like a lot of reverse proxy-to-SSL-backend tests failing with 502 errors due to the proxy failing to negotiate an SSL connection to the backend: [Mon Feb 20 09:27:50 2006] [info] SSL Library Error: 336130315 error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number (which is the kind of error you get when you try to negotiate SSL with something which doesn't speak SSL; perhaps the wrong backend is being selected?) joe
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On Mon, Feb 20, 2006 at 10:53:27AM +0100, Plüm, Rüdiger, VIS wrote: Von: Joe Orton New Revision: 378032 URL: http://svn.apache.org/viewcvs?rev=378032view=rev Log: *) mod_proxy: Fix KeepAlives not being allowed and set to backend servers. PR38602. [Ruediger Pluem, Jim Jagielski] Also, document previous patch: *) Correctly initialize mod_proxy workers, which use a combination of local and shared datasets. Adjust logging to better trace usage. PR38403. [Jim Jagielski] This one seems to have broken the proxy tests again, failed tests are: ... if I svn up -r378031 modules/proxy this goes away. This Hmm. Given http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/ajax/[EMAIL PROTECTED] this is really weird. Maybe another change that happened after r378032? Maybe Jim's build didn't include mod_ssl? r378032 is the most recent change to modules/proxy. joe
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On Mon, Feb 20, 2006 at 11:30:02AM +0100, Plüm, Rüdiger, VIS wrote: -Ursprüngliche Nachricht- Von: Joe Orton http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox /ajax/[EMAIL PROTECTED] this is really weird. Maybe another change that happened after r378032? Maybe Jim's build didn't include mod_ssl? r378032 is the most recent change to modules/proxy. Maybe. Got another idea in the meantime. Maybe it is because we use the same socket to a backend, but we create a new conn_rec each time for this backend and thus try to reinit the existing ssl connection. Could you please give the following patch a try? That fixed most of the failures, there are still two left: t/ssl/proxyok 61/172# Failed test 62 in t/ssl/proxy.t at line 112 fail #2 t/ssl/proxyok 112/172# Failed test 113 in /local/httpd/pf-trunk/blib/lib/Apache/TestCommonPost.pm at line 131 fail #102 t/ssl/proxyFAILED tests 62, 113 Failed 2/172 tests, 98.84% okay I found only one pertinent error message in the log: [Mon Feb 20 10:40:08 2006] [debug] proxy_util.c(2118): proxy: HTTP: connection complete to 127.0.0.1:8529 (localhost.localdomain) [Mon Feb 20 10:40:08 2006] [error] (32)Broken pipe: proxy: pass request body failed to 127.0.0.1:8529 (localhost.localdomain) [Mon Feb 20 10:40:08 2006] [error] (32)Broken pipe: proxy: pass request body failed to 127.0.0.1:8529 (localhost.localdomain) from 127.0.0.1 () ...perhaps a persistent connection being closed by the backend, and the proxy only finding out about this half way through sending a request body? Hard to handle that in the proxy - probably best to just ungracefully terminate the connection to the client in that case, it should then resend appropriately too. joe
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On Mon, Feb 20, 2006 at 12:52:31PM +0100, Plüm, Rüdiger, VIS wrote: I found only one pertinent error message in the log: [Mon Feb 20 10:40:08 2006] [debug] proxy_util.c(2118): proxy: HTTP: connection complete to 127.0.0.1:8529 (localhost.localdomain) But this is HTTP to the backend isn't it? Yup - this test covers both http/ssl to the backend with http/ssl to the client, omitting the http-http case. [Mon Feb 20 10:40:08 2006] [error] (32)Broken pipe: proxy: pass request body failed to 127.0.0.1:8529 (localhost.localdomain) [Mon Feb 20 10:40:08 2006] [error] (32)Broken pipe: proxy: pass request body failed to 127.0.0.1:8529 (localhost.localdomain) from 127.0.0.1 () ...perhaps a persistent connection being closed by the backend, and the proxy only finding out about this half way through sending a request Also my guess. See also point 2 in http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/ajax/[EMAIL PROTECTED] Is it possible to increase the keepalive timeout temporarily for a test run? This would give a valuable hint? Yup, that fixes the failures, though the test is hanging for a while in the two places it was failing before. There are a *lot* of these weird 408 (HTTP_REQUEST_TIME_OUT) errors in the access_log, I notice - it looks like these were happening even before the proxy changes: 127.0.0.1 - - [20/Feb/2006:13:17:44 +] - 408 223 127.0.0.1 - - [20/Feb/2006:13:17:45 +] POST /eat_post HTTP/1.1 200 6 127.0.0.1 - - [20/Feb/2006:13:17:45 +] POST /eat_post HTTP/1.0 200 6 127.0.0.1 - - [20/Feb/2006:13:17:45 +] - 408 223 that code is only generated new request parsing code. Brian? body? Hard to handle that in the proxy - probably best to just ungracefully terminate the connection to the client in that case, it should then resend appropriately too. Yes, if get into this situation I guess we have no better chance. But that should already be the reaction to this kind of situation. Ok, I see that this is currently not the case (at least not if we fail during sending the reponse). The following patch should sent a 502 to the client in such cases and close the connection to the client: That's not what I meant. This is a bit tricky. Here's a restatement of the problem, just to make sure we're talking about the same thing: If a client reuses a connection to an HTTP server which has been held open persistently, there is a risk that the connection may be closed either beforehand or simultaneously to when the client tries to reuse it. So, it's quite normal to start sending a POST request with a body, and then half way through sending the request body, you get an TCP RST or a FIN. HTTP/1.1 clients have logic to open a new connection and resend the request if that happens. If the client in this case is the proxy, then the proxy may not be able to resend the whole request body if it is streaming it through and may already have discarded the first half by the time it sees the RST/FIN/. If the proxy has the correct logic then this can be handled correctly. In handling a request which includes a body (and the body will not/cannot be cached entirely in memory): a) if the connection to the client has been kept open persistently (i.e. r-connection-keepalives 1), then it is safe to forward the request on a connection to the backend which has itself been held open persistently. If the backend connection is closed whilst sending the request/body, then the connection to the client should also immediately be closed without sending a response. b) if the connection to the client has *not* been kept open persistently, then it is only safe to forward a request which includes a request body on a newly opened connection to the backend. (since such a connection is not vulnerable to a persistent connection timeout) Does that make sense? Regards, joe
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On 02/20/2006 02:51 PM, Joe Orton wrote: That's not what I meant. This is a bit tricky. Here's a restatement of the problem, just to make sure we're talking about the same thing: If a client reuses a connection to an HTTP server which has been held open persistently, there is a risk that the connection may be closed either beforehand or simultaneously to when the client tries to reuse it. So, it's quite normal to start sending a POST request with a body, and then half way through sending the request body, you get an TCP RST or a FIN. HTTP/1.1 clients have logic to open a new connection and resend the request if that happens. Do you have any examples for such clients? All common browsers? If the client in this case is the proxy, then the proxy may not be able to resend the whole request body if it is streaming it through and may already have discarded the first half by the time it sees the RST/FIN/. If the proxy has the correct logic then this can be handled correctly. In handling a request which includes a body (and the body will not/cannot be cached entirely in memory): Thanks for explanation. Now I understand what you mean a) if the connection to the client has been kept open persistently (i.e. r-connection-keepalives 1), then it is safe to forward the request on a connection to the backend which has itself been held open persistently. If the backend connection is closed whilst sending the request/body, then the connection to the client should also immediately be closed without sending a response. Ok. Possibly tricky to implement. From my first view it would require the ap_http_header_filter to be removed, r-connection-keepalive set to AP_CONN_CLOSE and an eos sent down the chain, but I am neither sure that this is 1. RFC compliant 2. Good code b) if the connection to the client has *not* been kept open persistently, then it is only safe to forward a request which includes a request body on a newly opened connection to the backend. (since such a connection is not vulnerable to a persistent connection timeout) In principle we know too late if the request contains a body or not (checked in ap_proxy_http_request, but we would need to know in ap_proxy_connect_backend). We could check for C-L or T-E in headers_in as a marker for a possible request body. But this could be used by evil / bad clients to close pooled backend connections without actually sending a request body. Or is this assumption to paranoid? Regards Rüdiger
Re: svn commit: r378032 - in /httpd/httpd/trunk: CHANGES modules/proxy/mod_proxy_http.c modules/proxy/proxy_util.c
On 02/20/2006 11:50 AM, Joe Orton wrote: Maybe. Got another idea in the meantime. Maybe it is because we use the same socket to a backend, but we create a new conn_rec each time for this backend and thus try to reinit the existing ssl connection. Could you please give the following patch a try? Meanwhile I committed a better version of the patch as r379237. http://svn.apache.org/viewcvs?rev=379237view=rev If you have some cycles left could you please test? (I know that I should really get the test framework running on my box. First thing on my TODO list) That fixed most of the failures, there are still two left: Regards Rüdiger
