Hello Stéphane, On Tue, May 09, 2017 at 09:37:35AM +0200, Stéphane Claret wrote: > Good morning, > > In the current version (1.7.5), haproxy documentation states that > compression is disabled if the response is not chunked and does not include > a Content-length. > > Unfortunately, it seems a bit limitating because some Java application > servers (Jetty for sure, and some netty-based frameworks) never use chunked > mode together with Connection: close.
Oh that reminds me some old memories, I seem to remember that it used to consider that close==1.0 and keep-alive==1.1 and since there's no chunk in 1.0 it does not emit them. > For example, when requesting a payload with Connection: close set on the > client side, you'll typically receive the following response header. > > Date: Tue, 09 May 2017 06:07:17 GMT > Content-Type: application/json > Connection: close > Server: Jetty(9.3.12.v20160915) > > Content-length is included only in the particular case where the response > body fits in the server buffer, which is often not the case for large, > dynamically generated content, and this is causing haproxy to disable > compression. Of course. > According to discussions on the jetty github, when Connection: close is > present in the response, it's acceptable to simply let the client know that > the response is complete by closing the connection on the server side. It's acceptable but it's a bad practice because there is no way for the client to tell whether the download is complete or not. That's why chunks were added in HTTP/1.1. Truncated responses are very common due to timeouts on random parts of the infrastructure and when a client detects that the connection was closed without advertised content-length, the best it can do is consider that by definition the object was complete. However it can for example decide not to cache it or to always revalidate precisely to ensure that any truncated object could later be fixed. With chunks there is no doubt at all, the client knows that the object is not complete or that it is complete, and there is no risk to cache a truncted response for example. > On haproxy side, I have noticed that you can set option http-pretend-keep > alive to work this issue around, as this will remove "connection: close" > from the request header, thus allowing jetty to use chunked transfer > encoding. Yes, I think we implemented it for this specific purpose. > But given that Jetty is not alone in this case, I am wondering whether it > would be technically possible for haproxy to enable compression in the > following scenario (According to the response headers) : > > 1) HTTP 1.1 is used > 2) "Transfer-Encoding" is NOT chunked > 3) "Content-Length" is NOT specified > 4) "Connection : close" IS specified (expecting EOF framed response) Technically it's perfectly possible, you just have to remove the test doing this. But then you'll get some random, undebuggable complaints from your users because this would change the semantics : instead of delivering an object till the connection closes, we would turn it to chunks and really make the client believe that the end was reached, which is wrong as we don't know either. And *this* is a real problem because we'd deliver truncated objects saying "believe me, it's complete". Before we made this choice, I had an idea consisting in not sending the final 0-length chunk in this case, but many clients consider this as a proof of truncation and immediately remove the object or report a page load error, so that was not acceptable at all. The only possibly doable thing would be to deliver the chunks on the fly without using the chunked encoding. This way the client would have no way to know if the object is complete or not and that might be better than nothing. However some clients might consider that the EOF in the gzip stream is a proof of completeness, while it's not. Quite frankly, I consider it really problematic to start to deliver transformed objects and hiding problems to clients, pretending everything went well. I prefer to find good ways to ensure these servers deliver properly delimited objects instead of cheating on the response. Are you currently facing issues with real world servers that cannot be worked around using "http-pretend-keepalive" ? I'd rather work on this. And we have one or two Jetty developers here on the list if that can help there. Regards, Willy
