Re: HTTP/1.1 to clients in v3.2
mån 2010-08-16 klockan 14:08 -0600 skrev Alex Rousskov: > Should we try implement what you described then? In summary: > >- If the next hop is known to be 1.1, send a chunked request. >- Otherwise, try to accumulate the full body, with a timeout. > Send 411 if we timed out while accumulating. > > At first, the accumulation will still happen on the client side, like > today. Eventually, the accumulation code can be moved to the server side. Yes. Suggested implementaiton order: 1. Forward of chunked requests. 2. Next hop version cache, rejecting with 411 if next hop not known to be 1.1 3. Delay 411 response condition a bit, buffering the request hoping to be able to dechunk instead. Respond with 411 on timeout or buffer full. 4. tuning knob to selectively assume next hop is 1.1 if unknown, enabling out of band knowledge of 1.1 capability via configuration. This should also include tuning for selectively disabling chunked forwarding enabling ban of broken nex-hops. 5. Option to add "Expect: 100-continue" on forwarded chunked requests when forced 1.1, with it's requirements on delaying forwarding and retrying without expectation if seeing a 417 in response or returning 411 if retrying is not possible at time of 417. Note that when Expect: 100-continue is used by the client and complete path is 1.1 then 3 should not really happen due to the client delaying it's transmission for some considerable amount of time. Here we SHOULD instead respond with 411 immediately to follow the 100 Continue expected flow model, enabling client fallback to a much shorter delay. In paralell to this it's also needed to deal with 1xx responses in an reasonable manner, especially 100 Continue. Without these it's hard to get the expected flow of events running. Regards Henrik
Re: HTTP/1.1 to clients in v3.2
On 08/16/2010 01:52 PM, Henrik Nordström wrote: mån 2010-08-16 klockan 13:30 -0600 skrev Alex Rousskov: Since Squid is a program and not a human being, we do need to hard-code a single default. Clearly, there will be ACLs to change the behavior, but if no options apply, we still need to do something. Yu have more-or-less said "no" to every option I listed :-). Correct me if I am wrong, but I sense that option #1 (always send chunked request) is the "least bad" default. I will try to implement that unless you stop me. I did not say no, just the complications each proposal involves.. The correct per spec is to reject with 411 unless next hop is known to be 1.1. Alternatively dechunk if we have already received the full body (by waiting a little, not sending 100 Continue) Should we try implement what you described then? In summary: - If the next hop is known to be 1.1, send a chunked request. - Otherwise, try to accumulate the full body, with a timeout. Send 411 if we timed out while accumulating. At first, the accumulation will still happen on the client side, like today. Eventually, the accumulation code can be moved to the server side. Is this the right path forward? Thank you, Alex. An interception client is arguably more likely to know the next hop capabilities because it thinks it is talking directly to that hop. I would argue the opposite. It's less likely to make a correct decision as it do not expect the proxy to be there messing with things so it quite likely will take the response version of the proxy as an indication of the origin server capabilities not caring to look for Via etc.. Similarly, we are less likely to be blamed for screwing things up if we just repeat what the intercepted client did. Until someone slaps us with the specifications. Regards Henrik
Re: HTTP/1.1 to clients in v3.2
mån 2010-08-16 klockan 13:30 -0600 skrev Alex Rousskov: > Since Squid is a program and not a human being, we do need to hard-code > a single default. Clearly, there will be ACLs to change the behavior, > but if no options apply, we still need to do something. > > Yu have more-or-less said "no" to every option I listed :-). Correct me > if I am wrong, but I sense that option #1 (always send chunked request) > is the "least bad" default. I will try to implement that unless you stop me. I did not say no, just the complications each proposal involves.. The correct per spec is to reject with 411 unless next hop is known to be 1.1. Alternatively dechunk if we have already received the full body (by waiting a little, not sending 100 Continue) > An interception client is arguably more likely to know the next hop > capabilities because it thinks it is talking directly to that hop. I would argue the opposite. It's less likely to make a correct decision as it do not expect the proxy to be there messing with things so it quite likely will take the response version of the proxy as an indication of the origin server capabilities not caring to look for Via etc.. > Similarly, we are less likely to be blamed for screwing things up if we > just repeat what the intercepted client did. Until someone slaps us with the specifications. Regards Henrik
Re: HTTP/1.1 to clients in v3.2
On 08/16/2010 01:17 PM, Henrik Nordström wrote: mån 2010-08-16 klockan 11:43 -0600 skrev Alex Rousskov: I am revisiting this issue in hope to enable HTTP/1.1 to clients. If Squid properly dechunks requests on the client side, what should happen to that dechunked request on the server side? Let's start with the most general case were we do _not_ know the origin server version and where there are no matching ACLs to control Squid behavior. I see several options: 1. Always send a chunked request (because the client did so and perhaps the client knows that the server is HTTP/1.1 and can handle a chunked request.) But the client doesn't really know.. it knows Squid. HTTP version, transfer encoding etc is hop-by-hop, not end-to-end. Well, there is Via, but not sent on 100 Continue for example. Specs says "MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant". But we should at least have a knob for tuning this. I don't really agree with the specs here. As you note if the client sends chunked then in reality it's quite likely the server supports chunked as the client most likely have some out of band information about the server capabilities. 2. Always send dechunked request with a Content-Length field, after buffering all the chunks and calculating content length (because we do not know whether the server is HTTP/1.1 and can handle a chunked request). Not really doable. Request may be huge and buffer space limited. I am entirely fine with dechunking and converting to content-length should we already have the data buffered. But not to use it as default mechanism. Limited buffering plays somewhat badly with 100 Continue. I am not very comfortable with sending a 411 after 100 Continue even if not strictly illegal. Very unlikely clients deal well with such situation as specs says the 100 Continue indicates explicitly that the origin server is willing to accept the body. 3. Always add an "Expect: 100-continue" if not already there, send the headers, and then pick option #1 or #2, depending on the server version if we get a 100 Continue response. This should work out reasonably well I think, but adds a fair amount of complexity. If we get 100 Continue then the server is supposed to be 1.1. But MAY be an intermediary seding 100 Continue (not strictly legal..) If we add 100-continue then we must also retry without 100-continue if receiving a 417. Which also means that we need to buffer the request while it's forwarded, which risks running into the same limited buffering issue, requiring us to send a 411 response if buffer space has been exceeded and receiving a 417 response. Any other options? I don't think a single hard answer can be given. More likely a mix and some knobs to tune to real world brokenness will be needed. But in the short run blindly forwarding quite liekly works. Since Squid is a program and not a human being, we do need to hard-code a single default. Clearly, there will be ACLs to change the behavior, but if no options apply, we still need to do something. Yu have more-or-less said "no" to every option I listed :-). Correct me if I am wrong, but I sense that option #1 (always send chunked request) is the "least bad" default. I will try to implement that unless you stop me. Should the choice depend on whether we are a direct or interception proxy? Not really. End result is pretty much the same, except that the client have even less knowledge about the state of things when being intercepted. An interception client is arguably more likely to know the next hop capabilities because it thinks it is talking directly to that hop. Similarly, we are less likely to be blamed for screwing things up if we just repeat what the intercepted client did. Thank you, Alex.
Re: HTTP/1.1 to clients in v3.2
mån 2010-08-16 klockan 11:43 -0600 skrev Alex Rousskov: > I am revisiting this issue in hope to enable HTTP/1.1 to clients. If > Squid properly dechunks requests on the client side, what should happen > to that dechunked request on the server side? Let's start with the most > general case were we do _not_ know the origin server version and where > there are no matching ACLs to control Squid behavior. > > I see several options: > > 1. Always send a chunked request (because the client did so and perhaps > the client knows that the server is HTTP/1.1 and can handle a chunked > request.) But the client doesn't really know.. it knows Squid. HTTP version, transfer encoding etc is hop-by-hop, not end-to-end. Well, there is Via, but not sent on 100 Continue for example. Specs says "MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant". But we should at least have a knob for tuning this. I don't really agree with the specs here. As you note if the client sends chunked then in reality it's quite likely the server supports chunked as the client most likely have some out of band information about the server capabilities. > 2. Always send dechunked request with a Content-Length field, after > buffering all the chunks and calculating content length (because we do > not know whether the server is HTTP/1.1 and can handle a chunked request). Not really doable. Request may be huge and buffer space limited. I am entirely fine with dechunking and converting to content-length should we already have the data buffered. But not to use it as default mechanism. Limited buffering plays somewhat badly with 100 Continue. I am not very comfortable with sending a 411 after 100 Continue even if not strictly illegal. Very unlikely clients deal well with such situation as specs says the 100 Continue indicates explicitly that the origin server is willing to accept the body. > 3. Always add an "Expect: 100-continue" if not already there, send the > headers, and then pick option #1 or #2, depending on the server version > if we get a 100 Continue response. This should work out reasonably well I think, but adds a fair amount of complexity. If we get 100 Continue then the server is supposed to be 1.1. But MAY be an intermediary seding 100 Continue (not strictly legal..) If we add 100-continue then we must also retry without 100-continue if receiving a 417. Which also means that we need to buffer the request while it's forwarded, which risks running into the same limited buffering issue, requiring us to send a 411 response if buffer space has been exceeded and receiving a 417 response. > Any other options? I don't think a single hard answer can be given. More likely a mix and some knobs to tune to real world brokenness will be needed. But in the short run blindly forwarding quite liekly works. > Should the choice depend on whether we are a direct > or interception proxy? Not really. End result is pretty much the same, except that the client have even less knowledge about the state of things when being intercepted. Regards Henrik
