Re: mod_deflate Vary header tweak
On tis, 2008-04-29 at 09:42 +0200, André Malo wrote: > Just to be exact - it *might* vary, depending on how no-gzip is set. But then most likely not based on Accept-Encoding but other headers such as User-Agent or the source IP... In any event I fully agree that it's then the responsibility of whatever that set the no-gzip flag to also add a proper Vary attribution to the response. Only if no-gzip is set unconditionall should Vary not be added by the one setting no-gzip. But it's acceptable (even if not 100% correct) to not add Vary when setting no-gzip if one then accepts that the uncompressed variant ay get sent to more clients by downstream cache servers. Regards Henrik
Re: mod_deflate Vary header tweak
* Nick Kew wrote: > On Mon, 28 Apr 2008 15:10:14 -0400 > > "Brian J. France" <[EMAIL PROTECTED]> wrote: > > I would like to propose a change to mod_deflate that would still > > send the Vary header if the request is flagged with no-gzip or > > gzip-only- text/html. > > But if no-gzip is set, then the response will not be compressed, > regardless of the Accept-Encoding header. So it doesn't vary. Just to be exact - it *might* vary, depending on how no-gzip is set. However, this is still not within the scope of mod_deflate. Thatswhy we have provided this sample configuration in the mod_deflate docs together with mod_headers. nd
Re: mod_deflate Vary header tweak
On Mon, 28 Apr 2008 15:10:14 -0400 "Brian J. France" <[EMAIL PROTECTED]> wrote: > I would like to propose a change to mod_deflate that would still > send the Vary header if the request is flagged with no-gzip or > gzip-only- text/html. But if no-gzip is set, then the response will not be compressed, regardless of the Accept-Encoding header. So it doesn't vary. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
mod_deflate Vary header tweak
I would like to propose a change to mod_deflate that would still send
the Vary header if the request is flagged with no-gzip or gzip-only-
text/html.
I think the checks for Content-Range and Content-Encoding should be
done, then the Vary header should be set and then the checks for no-
gzip and gzip-only-text/html should be done.
Thoughts?
Brian
Index: mod_deflate.c
===
--- mod_deflate.c (revision 652071)
+++ mod_deflate.c (working copy)
@@ -422,34 +422,12 @@
return ap_pass_brigade(f->next, bb);
}
-/* some browsers might have problems, so set no-gzip
- * (with browsermatch) for them
- */
-if (apr_table_get(r->subprocess_env, "no-gzip")) {
-ap_remove_output_filter(f);
-return ap_pass_brigade(f->next, bb);
-}
-
/* We can't operate on Content-Ranges */
if (apr_table_get(r->headers_out, "Content-Range") != NULL) {
ap_remove_output_filter(f);
return ap_pass_brigade(f->next, bb);
}
-/* Some browsers might have problems with content types
- * other than text/html, so set gzip-only-text/html
- * (with browsermatch) for them
- */
-if (r->content_type == NULL
- || strncmp(r->content_type, "text/html", 9)) {
-const char *env_value = apr_table_get(r->subprocess_env,
- "gzip-only-text/
html");
-if ( env_value && (strcmp(env_value,"1") == 0) ) {
-ap_remove_output_filter(f);
-return ap_pass_brigade(f->next, bb);
-}
-}
-
/* Let's see what our current Content-Encoding is.
* If it's already encoded, don't compress again.
* (We could, but let's not.)
@@ -500,6 +478,28 @@
*/
apr_table_mergen(r->headers_out, "Vary", "Accept-Encoding");
+/* some browsers might have problems, so set no-gzip
+ * (with browsermatch) for them
+ */
+if (apr_table_get(r->subprocess_env, "no-gzip")) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f->next, bb);
+}
+
+/* Some browsers might have problems with content types
+ * other than text/html, so set gzip-only-text/html
+ * (with browsermatch) for them
+ */
+if (r->content_type == NULL
+ || strncmp(r->content_type, "text/html", 9)) {
+const char *env_value = apr_table_get(r->subprocess_env,
+ "gzip-only-text/
html");
+if ( env_value && (strcmp(env_value,"1") == 0) ) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f->next, bb);
+}
+}
+
/* force-gzip will just force it out regardless if the browser
* can actually do anything with it.
*/
Re: mod_deflate Vary header
On Tue, 8 Nov 2005 [EMAIL PROTECTED] wrote: Igor Sysoev wrote Actually, with MSIE 5.5+ appearance the chances that client can not decompress the response from downstream cache have increased. If MSIE 5.5 is configured to work via proxy with HTTP/1.0, then MSIE will never send "Accept-Encoding" header, and it would refuse the compressed content. You are right on the first part. If you don't have the "Use HTTP/1.1 for Proxies" checkbox on then no "Accept-Encoding:" header would be sent... ( principle of least astonishment applies here ). ...however... I think you will discover on closer inspection that the second part is not true. Even if MSIE 5.x doesn't send an "Accept-Encoding: gzip" header... it KNOWS it can decompress and will do so for any response that shows up with "Content-encoding: gzip" regardless of whether it sent an "Accept-Encoding: gzip" header. I've just checked MSIE 6.0 configured to work via proxy with HTTP/1.0 (default settings). I've preloaded into the proxy the compressed pages using Firefox. Then I've asked the same pages from MSIE. MSIE has showed 1) compressed pages for "/page.html" requests 2) and the "Save file" message box for the "/dir/" requests. My mod_deflate allows optionally to add "Vary" header to compressed response only. So you still run the risk of only getting one variant 'stuck' in a downstream cache. If the uncompressed version goes out at 'start of day' with no "Vary:" then the uncompressed version can get 'stuck' where you don't want it until it actually expires. I prefer to use compressing as safe as possible. So the default settings of my mod_deflate and my new lightweight server named nginx are: 1) compress "text/html" only; 2) compress HTTP/1.1 requests only; 3) do not compress responses for the proxied requests at all. And of course, mod_deflate and nginx have many directives to change this behavior: for example, they may compress responses for proxied requests if responses have "Cache-Control" header with the "private", "no-store", "no-cache" values. Igor Sysoev http://sysoev.ru/en/
Re: mod_deflate Vary header
> Igor Sysoev wrote > > Actually, with MSIE 5.5+ appearance the chances that client can not > decompress the response from downstream cache have increased. > If MSIE 5.5 is configured to work via proxy with HTTP/1.0, then > MSIE will never send "Accept-Encoding" header, and it would refuse > the compressed content. You are right on the first part. If you don't have the "Use HTTP/1.1 for Proxies" checkbox on then no "Accept-Encoding:" header would be sent... ( principle of least astonishment applies here ). ...however... I think you will discover on closer inspection that the second part is not true. Even if MSIE 5.x doesn't send an "Accept-Encoding: gzip" header... it KNOWS it can decompress and will do so for any response that shows up with "Content-encoding: gzip" regardless of whether it sent an "Accept-Encoding: gzip" header. It's part of the wild and whacky world of browsers. Netscape exhibits similar behavior. Rather than rejecting something when they know they can do it, regardless of protocol 'envelope'... they just go ahead and do it anyway. There were ( are ) versions of Netscape, MSIE and Opera that were capable of decompressing "Content-encoding: gzip" even BEFORE they added any HTTP/1.1 support at all. Go figure. > Actually, MSIE 5.5+ will cache the response with any "Vary" header > if it also has "Content-Encoding: gzip" or "Content-Encoding: deflate" > headers. But it refuses to cache if the response has no > "Content-Encoding: gzip" header. Thanks for pointing that out. You are right. I didn't make that exception clear in last message. If response has "Content-Encoding: gzip ( or deflate )" then It HAS to cache it. It will even do so beyond all "Cache-Control:" directives. Both MSIE and Netscape actually USE their own local cache to perform the actual decompression and, as such, will ALWAYS write the compressed data to disk first regardless of any "Cache-Control:" directives or "Vary:" headers or anything else, for that matter. MSIE will end up with only the decompressed version in the cache but Netscape will end up with TWO copies of the response in it's local cache... the original compressed version and the decompressed version. Only problem with Netscape is that it then goes brain-dead and forgets that it has 2 cached copies of the same response and sometimes tries to actually PRINT the compressed version of the response out of it's local cache. Not sure they ever solved that one. "Vary:" really doesn't hold much meaning for end-point client caches, anyway, so it's curious that it actually affects browser caching behavior one way or the other. "Vary:" is/was mostly meant to just help intermediate caches figure out "what to send" to actual end-point agents ( the ones ASKING for the right response ). I can't think of many HTTP response headers that would/should matter a hoot to the end-point browser as far as "Vary:" goes once it has a response to a request. It's all about 'freshness' from the end-point cache's point of view. It's the browser that expects someone else to "Vary:" the response based on the REQUEST headers it sent. Only reason I ever heard for MSIE deciding to never cache any response with a "Vary:" header ( Other than compressed responses ) goes back to some huge bug they had to fix that was being caused by a "Vary:" header and, rather than actually add true "Vary:" support, the quick fix was to just reject everything. Better to point the blame upstream than ( God forbid ) show the user the wrong thing. I believe Netscape used the same approach. If you can't do ALL of the "Vary:" rules then safest thing to do is just reject it all and let someone ( or something ) else worry about getting you the right content. This was also SQUID's philosophy with regards to "Vary:" in the not-too-distant past, before they made any attempt to support multiple variants at all. SQUID would never bother to cache ANYTHING that had a "Vary:" header on it. > My mod_deflate allows optionally to add "Vary" header to compressed > response only. So you still run the risk of only getting one variant 'stuck' in a downstream cache. If the uncompressed version goes out at 'start of day' with no "Vary:" then the uncompressed version can get 'stuck' where you don't want it until it actually expires. Yours... Kevin Kiley
Re: mod_deflate Vary header
On Fri, 4 Nov 2005 [EMAIL PROTECTED] wrote: This has been discussed many times before and no one seems to understand what the fundamental problem is. It is not with the servers at all, it is with the CLIENTS. What both of you are saying is true... whether you "Vary:" on "Content-encoding" and/or "User-agent" or not... there is a risk of getting the wrong content ( compressed versus uncompressed ) "stuck" in a downstream cache. It is less and less likely these days that the cache will receive a request from a client that CANNOT "decompress", but still possible. Handling requests from clients that cannot decompress have become ( at long last ) the "fringe" cases but are no less important than ever. Actually, with MSIE 5.5+ appearance the chances that client can not decompress the response from downstream cache have increased. If MSIE 5.5 is configured to work via proxy with HTTP/1.0, then MSIE will never send "Accept-Encoding" header, and it would refuse the compressed content. This is why my mod_deflate for Apache 1.3 by default does not compress response for proxied requests. Microsoft Internet Explorer ( all versions ) will REFUSE to cache anything locally if it shows up with any "Vary:" headers on it. Period. End of sentence. So you might think you are doing your downstream clients a favor by tacking on a "Vary:" header to the compressed data so it gets 'stored' somewhere close to them... but you would be wrong. If you don't put a "Vary:" header on it... MSIE will, in fact, cache the compressed response locally and life is good. The user won't even come back for it until it's expired on their own hard drive or they clear their browser cache. However... if you simply add a "Vary:" header to the same compressed response... MSIE now refuses to cache that response at all and now you create a "thundering herd" scenario whereby the page is never local to the user for any length of time and each "forward" or "back" button hit causes the client to go upstream for the page each and every time. Even if there is a cache nearby you would discover that the clients are nailing it each and every time for the same page just because it has a "Vary:" header on it. I believe Netscape has the same problem(s). I don't use Netscape anymore. Anyone know for sure if Netscape actually stores "variants" correctly in local browser cache? Actually, MSIE 5.5+ will cache the response with any "Vary" header if it also has "Content-Encoding: gzip" or "Content-Encoding: deflate" headers. But it refuses to cache if the response has no "Content-Encoding: gzip" header. My mod_deflate allows optionally to add "Vary" header to compressed response only. Igor Sysoev http://sysoev.ru/en/
Re: mod_deflate Vary header
Hi, > I did not talk about traffic, but bandwidth. I read that you're concerned > about server's bandwidth and so it would be good if the clients get > uncompressed content from the cache rather than compressed from the server. > The trick to achieve both is *to deliver uncompressed content* without > 'Vary'ing the response. ... which, though, would increase loading time of the resource and traffic on the server if most clients do support compression and only few shared caches ever get hit by requests for uncompressed content, no? (All that assuming at least basic Vary: support in caches, of course. Without it, things practically probably are a bit different ...) Cya, Florian
Re: mod_deflate Vary header
* Florian Zumbiehl wrote: > Hi, > > > > The whole point is I don't do compression because of any _clients'_ > > > connections, but because of the _server's_ connection! If the > > > server's connection usually is far slower than the client's > > > connection (like with a server behind a V.90 modem, which would be > > > 33.6 kb/s upstream with considerable latency), it (1) would be faster > > > for the vast majority of clients to get an uncompressed copy from > > > their ISP's cache rather than a compressed one from the original > > > server and (2) even if it wasn't, it still might save traffic and > > > thus money for the server operator. > > > > If I do understand correctly, you say, that not compressing the content > > saves bandwidth? Well then turn off mod_deflate and you're done. No > > need to change anything here, IMHO. > > Could you possibly point out where you think you've read that in the > text quoted above? In case you just have a particular scenario in mind > where at least a large part of the caches that hold a copy of the > compressed version sooner or later get a request for the uncompressed > version: How about considering a scenario where my suggestion could be > sensible rather than specifically making one up where it is not? > > Plus, I obviously listed possible traffic reduction at the server only > as a secondary advantage. I did not talk about traffic, but bandwidth. I read that you're concerned about server's bandwidth and so it would be good if the clients get uncompressed content from the cache rather than compressed from the server. The trick to achieve both is *to deliver uncompressed content* without 'Vary'ing the response. nd -- > Rätselnd, was ein Anthroposoph mit Unterwerfung zu tun hat... [...] Dieses Wort gibt so viele Stellen für einen Spelling Flame her, und Du gönnst einem keine einzige.-- Jean Claude und David Kastrup in dtl
Re: mod_deflate Vary header
Hi, > > The whole point is I don't do compression because of any _clients'_ > > connections, but because of the _server's_ connection! If the server's > > connection usually is far slower than the client's connection (like > > with a server behind a V.90 modem, which would be 33.6 kb/s upstream > > with considerable latency), it (1) would be faster for the vast majority > > of clients to get an uncompressed copy from their ISP's cache rather than > > a compressed one from the original server and (2) even if it wasn't, it > > still might save traffic and thus money for the server operator. > > If I do understand correctly, you say, that not compressing the content > saves bandwidth? Well then turn off mod_deflate and you're done. No need to > change anything here, IMHO. Could you possibly point out where you think you've read that in the text quoted above? In case you just have a particular scenario in mind where at least a large part of the caches that hold a copy of the compressed version sooner or later get a request for the uncompressed version: How about considering a scenario where my suggestion could be sensible rather than specifically making one up where it is not? Plus, I obviously listed possible traffic reduction at the server only as a secondary advantage. Cya, Florian
Re: mod_deflate Vary header
* Florian Zumbiehl wrote: > The whole point is I don't do compression because of any _clients'_ > connections, but because of the _server's_ connection! If the server's > connection usually is far slower than the client's connection (like > with a server behind a V.90 modem, which would be 33.6 kb/s upstream > with considerable latency), it (1) would be faster for the vast majority > of clients to get an uncompressed copy from their ISP's cache rather than > a compressed one from the original server and (2) even if it wasn't, it > still might save traffic and thus money for the server operator. If I do understand correctly, you say, that not compressing the content saves bandwidth? Well then turn off mod_deflate and you're done. No need to change anything here, IMHO. nd -- Gefunden auf einer "Webdesigner"-Seite: > Programmierung in HTML, XML, WML, CGI, FLASH < # André Malo # http://pub.perlig.de/ #
Re: mod_deflate Vary header
Hi, [...] > > To put this straight: I was thinking about web servers behind > > V.90/ISDN/ADSL lines where _that_ line usually will be the bottleneck > > on any connections to the outside world and about caching proxies > > in that outside world ... > > Yes, but if you do compression because some of your clients have low bandwith > connections (but are capable of receiving compressed pages) and access your > server via a proxy then not sending the Vary header can "poison" the cache in > a way > you do not want. Because if the client which causes the proxy to cache the > response > is not capable to receive compressed pages will cause the proxy to cache > *only* the > uncompressed version of the page. The whole point is I don't do compression because of any _clients'_ connections, but because of the _server's_ connection! If the server's connection usually is far slower than the client's connection (like with a server behind a V.90 modem, which would be 33.6 kb/s upstream with considerable latency), it (1) would be faster for the vast majority of clients to get an uncompressed copy from their ISP's cache rather than a compressed one from the original server and (2) even if it wasn't, it still might save traffic and thus money for the server operator. Cya, Florian
Re: mod_deflate Vary header
This has been discussed many times before and no one seems to understand what the fundamental problem is. It is not with the servers at all, it is with the CLIENTS. What both of you are saying is true... whether you "Vary:" on "Content-encoding" and/or "User-agent" or not... there is a risk of getting the wrong content ( compressed versus uncompressed ) "stuck" in a downstream cache. It is less and less likely these days that the cache will receive a request from a client that CANNOT "decompress", but still possible. Handling requests from clients that cannot decompress have become ( at long last ) the "fringe" cases but are no less important than ever. Microsoft Internet Explorer ( all versions ) will REFUSE to cache anything locally if it shows up with any "Vary:" headers on it. Period. End of sentence. So you might think you are doing your downstream clients a favor by tacking on a "Vary:" header to the compressed data so it gets 'stored' somewhere close to them... but you would be wrong. If you don't put a "Vary:" header on it... MSIE will, in fact, cache the compressed response locally and life is good. The user won't even come back for it until it's expired on their own hard drive or they clear their browser cache. However... if you simply add a "Vary:" header to the same compressed response... MSIE now refuses to cache that response at all and now you create a "thundering herd" scenario whereby the page is never local to the user for any length of time and each "forward" or "back" button hit causes the client to go upstream for the page each and every time. Even if there is a cache nearby you would discover that the clients are nailing it each and every time for the same page just because it has a "Vary:" header on it. I believe Netscape has the same problem(s). I don't use Netscape anymore. Anyone know for sure if Netscape actually stores "variants" correctly in local browser cache? Yours... Kevin Kiley In a message dated 11/4/2005 4:55:02 PM Central Standard Time, [EMAIL PROTECTED] writes: On 11/04/2005 07:36 AM, Florian Zumbiehl wrote: [..cut..] > >>Maybe I'm pessimistic, but I think, omitting the Vary header for >>uncompressed ressources will lead to "poisoned" caches, which statistically >>nearly always will request the uncompressed variant and so actually >>*add* load to your server's bandwidth. > > > Hu? Erm, could it be that you are thinking of load-reducing caches put > directly in front of the web server? In that case, I wonder how the web > server's bandwidth could be the bottleneck?! > > To put this straight: I was thinking about web servers behind > V.90/ISDN/ADSL lines where _that_ line usually will be the bottleneck > on any connections to the outside world and about caching proxies > in that outside world ... Yes, but if you do compression because some of your clients have low bandwith connections (but are capable of receiving compressed pages) and access your server via a proxy then not sending the Vary header can "poison" the cache in a way you do not want. Because if the client which causes the proxy to cache the response is not capable to receive compressed pages will cause the proxy to cache *only* the uncompressed version of the page. Regards Rüdiger
Re: mod_deflate Vary header
On 11/04/2005 07:36 AM, Florian Zumbiehl wrote: [..cut..] > >>Maybe I'm pessimistic, but I think, omitting the Vary header for >>uncompressed ressources will lead to "poisoned" caches, which statistically >>nearly always will request the uncompressed variant and so actually >>*add* load to your server's bandwidth. > > > Hu? Erm, could it be that you are thinking of load-reducing caches put > directly in front of the web server? In that case, I wonder how the web > server's bandwidth could be the bottleneck?! > > To put this straight: I was thinking about web servers behind > V.90/ISDN/ADSL lines where _that_ line usually will be the bottleneck > on any connections to the outside world and about caching proxies > in that outside world ... Yes, but if you do compression because some of your clients have low bandwith connections (but are capable of receiving compressed pages) and access your server via a proxy then not sending the Vary header can "poison" the cache in a way you do not want. Because if the client which causes the proxy to cache the response is not capable to receive compressed pages will cause the proxy to cache *only* the uncompressed version of the page. Regards Rüdiger
Re: mod_deflate Vary header
Hi, > Yes, that is the point. The Vary header describes which header(s) was used > to decide, which content actually would be delivered. That's what HTTP > specifies. To be exact, I'd say that HTTP specifies that it's about representation and not about content - which might even be the point here: The content stays the same, just the representation could change, but only in so far as the client certainly will be able to understand it. > > When the bottleneck is at the client's end, it's completely different, > > of course--which is why I'd consider a config option a good idea that > > would allow the sending of a Vary: Content-Encoding in case of > > uncompressed content to be disabled. > > For your scenario it would be more useful to adjust the local cache, > I think. (e.g. always request compressed content, decompress it and > deliver it to all the clients uncompressed). Sure - just that you usually don't have any admin access to the vast majority of the caches that might be accessing the web page you are publishing =:-) > Maybe I'm pessimistic, but I think, omitting the Vary header for > uncompressed ressources will lead to "poisoned" caches, which statistically > nearly always will request the uncompressed variant and so actually > *add* load to your server's bandwidth. Hu? Erm, could it be that you are thinking of load-reducing caches put directly in front of the web server? In that case, I wonder how the web server's bandwidth could be the bottleneck?! To put this straight: I was thinking about web servers behind V.90/ISDN/ADSL lines where _that_ line usually will be the bottleneck on any connections to the outside world and about caching proxies in that outside world ... Cya, Florian
Re: mod_deflate Vary header
* Florian Zumbiehl <[EMAIL PROTECTED]> wrote: > While configuring and testing my new and shiny Apache 2, I noticed that > mod_deflate sends a Vary: Content-Encoding header whenever a resource > could potentially be compressed, no matter whether it actually is > compressed in that particular response. That certainly should > work, but in situations where the server's bandwidth is the bottleneck, > wouldn't it be a good idea to omit that Vary: header when no compression > actually takes place? After all, any user agent should be able to cope > with uncompressed content - and when that is what is already "behind > the bottleneck", like in some local proxy cache, a Vary: Content-Encoding > header on it would cause a request from a user agent supporting > compression to cause a reload of the compressed version from the slow > server rather than delivering the already locally-available uncompressed > version. Yes, that is the point. The Vary header describes which header(s) was used to decide, which content actually would be delivered. That's what HTTP specifies. > When the bottleneck is at the client's end, it's completely different, > of course--which is why I'd consider a config option a good idea that > would allow the sending of a Vary: Content-Encoding in case of > uncompressed content to be disabled. For your scenario it would be more useful to adjust the local cache, I think. (e.g. always request compressed content, decompress it and deliver it to all the clients uncompressed). Maybe I'm pessimistic, but I think, omitting the Vary header for uncompressed ressources will lead to "poisoned" caches, which statistically nearly always will request the uncompressed variant and so actually *add* load to your server's bandwidth. nd
mod_deflate Vary header
Hi, While configuring and testing my new and shiny Apache 2, I noticed that mod_deflate sends a Vary: Content-Encoding header whenever a resource could potentially be compressed, no matter whether it actually is compressed in that particular response. That certainly should work, but in situations where the server's bandwidth is the bottleneck, wouldn't it be a good idea to omit that Vary: header when no compression actually takes place? After all, any user agent should be able to cope with uncompressed content - and when that is what is already "behind the bottleneck", like in some local proxy cache, a Vary: Content-Encoding header on it would cause a request from a user agent supporting compression to cause a reload of the compressed version from the slow server rather than delivering the already locally-available uncompressed version. When the bottleneck is at the client's end, it's completely different, of course--which is why I'd consider a config option a good idea that would allow the sending of a Vary: Content-Encoding in case of uncompressed content to be disabled. Cya, Florian
