Re: Issue with persistent http proxy backend connection
On 10/11/2006 11:09 PM, Jeff Trawick wrote: On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods it is not a good idea to regard a GET / HEAD with query parameters to be idempotent. Location /proxy/all_gets_are_idempotent SetEnv foomatic=1 /Location Sorry I don't get the point here. Could you please explain? (In case it isn't stated explicitly already) No such retry would be performed on the initial request sent on a connection, right? Yes, this should be only done if we get an existing connection from the pool. Regards Rüdiger
Re: Issue with persistent http proxy backend connection
On 10/12/06, Ruediger Pluem [EMAIL PROTECTED] wrote: On 10/11/2006 11:09 PM, Jeff Trawick wrote: On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods it is not a good idea to regard a GET / HEAD with query parameters to be idempotent. Location /proxy/all_gets_are_idempotent SetEnv foomatic=1 /Location Sorry I don't get the point here. Could you please explain? Allow the administrator to declare that GETs/HEADs are idempotent even if they have query args, with a few lines of code to check some envvar.
Re: Issue with persistent http proxy backend connection
On 10/11/2006 11:09 PM, Jeff Trawick wrote: On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: 3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter timeout set. This can give a hint when the backend will close its persistent connection due to a timeout. Unfortunately, our own hint in the response is not useful when sent by a child process that is being terminated due to excessive idle processes/threads or MaxRequestsPerChild, since we can break out of ap_process_connection right after telling the client they have N seconds to send the next request. This will happen enough to be I do not think that this matters all too much, because the backend closes the connection *immediately* after sending out the response. So the socket connection check on proxy side that is executed before reusing a connection will detect that this socket has been closed by the remote side. But yes, in theory their remains a race here if the proxy reuses the connection faster then the backend needs time to close the socket after sending the response. Regards Rüdiger
Re: Issue with persistent http proxy backend connection
Ruediger Pluem wrote: But yes, in theory their remains a race here if the proxy reuses the connection faster then the backend needs time to close the socket after sending the response. Does the problem exists with the current trunk as well? The way how trunk detects the closed backend has changed by not using read any more. Regards, Mladen.
Re: Issue with persistent http proxy backend connection
On 10/12/06, Ruediger Pluem [EMAIL PROTECTED] wrote: On 10/11/2006 11:09 PM, Jeff Trawick wrote: On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: 3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter timeout set. This can give a hint when the backend will close its persistent connection due to a timeout. Unfortunately, our own hint in the response is not useful when sent by a child process that is being terminated due to excessive idle processes/threads or MaxRequestsPerChild, since we can break out of ap_process_connection right after telling the client they have N seconds to send the next request. This will happen enough to be I do not think that this matters all too much, because the backend closes the connection *immediately* after sending out the response. So the socket connection check on proxy side that is executed before reusing a connection will detect that this socket has been closed by the remote side. But yes, in theory their remains a race here if the proxy reuses the connection faster then the backend needs time to close the socket after sending the response. Agreed. response sent, then FIN (lingering close), so client/proxy should be able to detect terminated connection immediately; it would only break with pipelining, which proxy doesn't support
Re: Issue with persistent http proxy backend connection
On 12.10.2006 13:26, Mladen Turk wrote: Ruediger Pluem wrote: But yes, in theory their remains a race here if the proxy reuses the connection faster then the backend needs time to close the socket after sending the response. Does the problem exists with the current trunk as well? The way how trunk detects the closed backend has changed by not using read any more. Yes, because of a race condition. The remote side could close the connection just after the check happened. The closing could even happen while the first data is on its way to the backend. So the first write(s) can succeed, because the FIN from the backend is on its way during this time. Nevertheless it is very good to have a reliable socket connection check (as on trunk) as this reduces the number of cases where this happens a lot and makes handling of such cases a lot easier. Regards Rüdiger
Re: Issue with persistent http proxy backend connection
Ruediger Pluem wrote: I do not think that this matters all too much, because the backend closes the connection *immediately* after sending out the response. So the socket connection check on proxy side that is executed before reusing a connection will detect that this socket has been closed by the remote side. But yes, in theory their remains a race here if the proxy reuses the connection faster then the backend needs time to close the socket after sending the response. +1 on the 'in theory' part, but seeing this in the real world is very very remote. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: Issue with persistent http proxy backend connection
Jeff Trawick wrote: On 10/12/06, Ruediger Pluem [EMAIL PROTECTED] wrote: On 10/11/2006 11:09 PM, Jeff Trawick wrote: On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods it is not a good idea to regard a GET / HEAD with query parameters to be idempotent. Location /proxy/all_gets_are_idempotent SetEnv foomatic=1 /Location Sorry I don't get the point here. Could you please explain? Allow the administrator to declare that GETs/HEADs are idempotent even if they have query args, with a few lines of code to check some envvar. If actually needed ( :) ) +1 on concept -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: Issue with persistent http proxy backend connection
tor 2006-10-12 klockan 13:19 +0200 skrev Ruediger Pluem: I do not think that this matters all too much, because the backend closes the connection *immediately* after sending out the response. To help this, perhaps there should be a check just before sending the response as well, and send Connection: close if it's likely this thread should terminate after this response? MaxRequestsPerChild certainly can be evaluated before the response is sent. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Issue with persistent http proxy backend connection
There is an issue with the proxy code that if a request is sent over a persistent backend connection (currently only looking at the http case, not sure if the same thing can happen for other backends like ajp and fastcgi) it could happen that this connection gets closed by the backend for timeout reasons after the is_connected check and before / while sending the request. For reference see e.g: http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/[EMAIL PROTECTED] http://mail-archives.apache.org/mod_mbox/httpd-dev/200602.mbox/[EMAIL PROTECTED] I had some virtual hackathon with Joe Orton via IRC on this topic and try to summarize our findings: 1. RFC 2616 does not allow clients to resend a non idempotent request in those cases without user interaction (8.1.4). It is concluded that the same applies to a proxy in this case. 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods it is not a good idea to regard a GET / HEAD with query parameters to be idempotent. 3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter timeout set. This can give a hint when the backend will close its persistent connection due to a timeout. If there is a non idempotent request to the proxy and there is not enough timeout time left on the backend connection, then close the pooled backend connection and forward the request to the backend via a new connection. The size of enough timeout time left is not defined yet and besides a good default value for it, there should be the possibility to change that via configuration. If the server does not sent the timeout parameter back in its reponse, it is regarded as 0. If there is a idempotent request, resent it if it failed because of a read / write failure to the backend *before* the first successful read from the backend. It is important that this is really a socket error. Simply checking for a failure of ap_proxy_http_request is not *enough*, because it should not be resent if it failed due to some other problem (e.g. parsing error). Regards Rüdiger
Re: Issue with persistent http proxy backend connection
On 10/11/06, Ruediger Pluem [EMAIL PROTECTED] wrote: There is an issue with the proxy code that if a request is sent over a persistent backend connection (currently only looking at the http case, not sure if the same thing can happen for other backends like ajp and fastcgi) it could happen that this connection gets closed by the backend for timeout reasons after the is_connected check and before / while sending the request. 1. RFC 2616 does not allow clients to resend a non idempotent request in those cases without user interaction (8.1.4). It is concluded that the same applies to a proxy in this case. A non-compliant behavior (i.e., not default Apache behavior) which would be useful would be to wait for 100-Continue before sending POST bodies. If connection is dropped before receiving 100-Continue, try on a new connection. 2. Although GET is mentioned to be idempotent in RFC 2616 (9.1.2) along with some other methods it is not a good idea to regard a GET / HEAD with query parameters to be idempotent. Location /proxy/all_gets_are_idempotent SetEnv foomatic=1 /Location 3. Sometimes servers (including httpd) include a keep-alive header in their response with the parameter timeout set. This can give a hint when the backend will close its persistent connection due to a timeout. Unfortunately, our own hint in the response is not useful when sent by a child process that is being terminated due to excessive idle processes/threads or MaxRequestsPerChild, since we can break out of ap_process_connection right after telling the client they have N seconds to send the next request. This will happen enough to be disturbing, though perhaps not often enough to quickly diagnose it. (Gosh, try turning off proxy keepalive and see if it gets better. If so, that's your permanent solution.) If there is a non idempotent request to the proxy and there is not enough timeout time left on the backend connection, then close the pooled backend connection and forward the request to the backend via a new connection. The size of enough timeout time left is not defined yet and besides a good default value for it, there should be the possibility to change that via configuration. If the server does not sent the timeout parameter back in its reponse, it is regarded as 0. If there is a idempotent request, resent it if it failed because of a read / write failure to the backend *before* the first successful read from the backend. It is important that this is really a socket error. Simply checking for a failure of ap_proxy_http_request is not *enough*, because it should not be resent if it failed due to some other problem (e.g. parsing error). (In case it isn't stated explicitly already) No such retry would be performed on the initial request sent on a connection, right?
