Re: Can an Apache module inject configuration in runtime?
On Tue, Jun 1, 2010 at 11:39, Andrew Godziuk and...@cloudaccess.net wrote: What I want to achieve is injecting new vhosts without Apache restart. Of course I'm aware that the changes would fully take effect after all workers have recycled, but for me - it's still better than a restart. Andrew, wouldn't you be better off with something like mod_vhost_alias, as-is or as leitmotiv? Propagating configuration changes to all workers is rather difficult, especially with pre-fork MPMs.
Re: Can an Apache module inject configuration in runtime?
On Tue, Jun 1, 2010 at 11:51 AM, Ben Noordhuis i...@bnoordhuis.nl wrote: Andrew, wouldn't you be better off with something like mod_vhost_alias, as-is or as leitmotiv? Propagating configuration changes to all workers is rather difficult, especially with pre-fork MPMs. Unfortunately, I need some suexec mechanism, currently I'm using mpm_itk, so I need some per-vhost settings, it's not available with mod_vhost_alias style dispatch-time modules. At least, as far as I understand how they work. -- Andrew
Re: Can an Apache module inject configuration in runtime?
On 1 Jun 2010, at 10:39, Andrew Godziuk wrote: I'm wondering if it's possible for an Apache module to change global config structures. Basically, no, not without a restart. Though there are some aspects of configuration you can change on the fly. What I want to achieve is injecting new vhosts without Apache restart. Of course I'm aware that the changes would fully take effect after all workers have recycled, but for me - it's still better than a restart. There are some similar modules around. Someone already said mod_vhost_alias, but I'd also point you at mod_vhost_dbd as an example that may be nearer to what you want if your needs are sufficiently complex to demand a new module. -- Nick Kew
Re: Fast by default
On 6/1/2010 3:30 AM, Bryan McQuade wrote: I had a conversation with a well known hosting provider recently and they told me they use the default Apache configuration for their shared hosting service. When I asked if they provide gzip as an option for their users, they said no, since it was not enabled by default. When I explained to them that enabling gzip has significant benefits for end users they were very interested in turning on gzip. This company just used the default Apache config, assuming that it was reasonably well tuned by default. You can claim that they're making bad, uninformed decisions, or whatever you want to, but the fact remains: some Apache users assume that the default config is a reasonably good config, and use it as-is. There are two classes of users out there: power users that want to tune every knob manually, and typical users that just want Apache to work out of the box. Apache developers on this list are part of the former group, which I assume is why the default Apache config is very simple. It's designed by power users, for power users. But there are non-power users out there. It would be nice if the Apache community provided configuration hints for these users. In 2010, IMO there is no good reason to have gzip disabled by default. Almost all websites enable it. There are a handful of prominent websites that do not. I've had conversations with a few of these sites. Most of them have not turned it on because they don't understand what it does, not because they don't have enough CPU. gzip has been used on the web now for well over 10 years. Only *very* old browsers, proxies, etc don't have perfect support for gzip. I can respect that the default httpd.conf is designed to be simple and minimalist. But it would be helpful to have an additional example configuration in svn trunk and as part of Apache releases, that enable things like mod_deflate. The current comments in httpd.conf explain that there are additional directives and you have been warned but IMO this is not very helpful or specific. We can do better. I propose providing an additional httpd.conf in the svn trunk and as part of future Apache releases that enables modules and directives that are commonly recommended on Apache performance tuning websites. This includes mod_deflate, mod_expires, etc. This will allow power users to continue to start with the current httpd.conf while typical users can just use the well optimized configuration. Hopefully this suggestion isn't too controversial. If there are concerns about some of the specific directives suggested in this thread, I'm sure we can work those out through discussion. Can we agree that it would be useful to provide an additional configuration file for non-power users that enables commonly recommended modules by default? +1 The PHP project does something similar, and so did MySQL (they might still - haven't done a source install there lately). One could point out that most out-of-the-box users (the linux ones, at least) use their distribution's binaries (which is another story) but I don't feel that that alone justifies not doing something like this. I propose that the recommended production deployment httpd.conf enable the features that we feel should be adopted by the internet community, as a whole, regardless of individual use cases, and we can put a prominent notice that it can and should be tuned by system administrators. In addition to folks who still use it out-of-the-box we'll also get lots of power-users looking at what we consider to be new and important features for the future of the community. I'd also like to see TLS/SNI as a recommended feature for this proposed conf. Issac
Re: Fast by default
On 01.06.2010 07:19, Jerome Renard wrote: In 2010, IMO there is no good reason to have gzip disabled by default. Almost all websites enable it. There are a handful of prominent websites that do not. I've had conversations with a few of these sites. Most of them have not turned it on because they don't understand what it does, not because they don't have enough CPU. gzip has been used on the web now for well over 10 years. Only *very* old browsers, proxies, etc don't have perfect support for gzip. I remember having a lot of troubles with some combinations of Squid + IE (6 or so) where the compressed content was just never gunziped. Was I the only one to get such problems ? There are still problems with caching proxies, and the behaviour of mod_deflate has changed several times, even in the 2.2.x lifetime. There was a lot of discussion about it. As a starter you can read my last comment in https://issues.apache.org/bugzilla/show_bug.cgi?id=49358 and then follow the links given there. In short: mod_deflate uses Content-Encoding instead of Transfer-Encoding which is supported by the Browsers, but leads to tricky problems when chosing the right ETag (content equivalence, cache revalidation requests). Regards, Rainer
Re: Fast by default
On 01 Jun 2010, at 2:30 AM, Bryan McQuade wrote: I had a conversation with a well known hosting provider recently and they told me they use the default Apache configuration for their shared hosting service. When I asked if they provide gzip as an option for their users, they said no, since it was not enabled by default. When I explained to them that enabling gzip has significant benefits for end users they were very interested in turning on gzip. This company just used the default Apache config, assuming that it was reasonably well tuned by default. You can claim that they're making bad, uninformed decisions, or whatever you want to, but the fact remains: some Apache users assume that the default config is a reasonably good config, and use it as-is. The very definition of tuned means tailored for your local setup. The default httpd configuration works reasonably well out the box. It is only when your site has special needs that it should start changing the setup, and the site should understand what their needs are and whether it is appropriate to turn it on. Zooming into mod_deflate, mod_deflate only makes sense if you have the CPU to support it. If you don't have enough CPU support (think virtualised hosts), mod_deflate will be a performance drag, not a boost. Typically, you would want to front a mod_deflate with an HTTP cache, such as mod_cache (or equivalent). Here mod_cache only makes sense if you have the disk space to support it, and there is no real one-size-fits-all cache setup. The next problem is that you only want to enable mod_deflate on compressible content - that means not images for most people, but might not be. Again, not every site has the same content, and therefore not every site has the same setup for mod_delate. This said, our default config is 15 years old, and attempts to disable deflate for browsers that don't support it, like Netscape 4. Unless there are modern browsers that have broken protocol support for transfer encoding, these obsolete examples need to be removed. Regards, Graham --
Can an Apache module inject configuration in runtime?
Hi, First of all, hi to the Apache developers, I'm new to this list - nice to meet you all. I'm wondering if it's possible for an Apache module to change global config structures. What I want to achieve is injecting new vhosts without Apache restart. Of course I'm aware that the changes would fully take effect after all workers have recycled, but for me - it's still better than a restart. I've written an Apache module before, but the configuration is an unknown land to me. While reading config.c, I noticed that a function called ap_build_config() could be helpful, but how do I call it to do what I need? Is it at all possible? -- Andrew
[PROPOSAL] Keep-Alive while exiting (Was: Unclean process shutdown in event MPM?)
Situation: worker or event MPM. Process shutdown due to: - MaxRequestsPerChild - MaxSpareThreads - Graceful stop or graceful restart When an httpd child process shuts down due to the above conditions, it doesn't respect existing Keep-Alive connections. When the previous response signalled keep-alive to the client and no next request was received, the connection will be terminated. This is especially noticeable for event, though I can reproduce for worker as well. Based on a patch by Jeff http://mail-archives.apache.org/mod_mbox/httpd-dev/200711.mbox/%3ccc67648e0711130530h45c2a28ctcd743b2160e22...@mail.gmail.com%3e who implemented KeepAliveWhileExiting I worked on a patch for trunk (and another one for 2.2.x) which seems to solve the problem in the above cases, maybe also for prefork, but I didn't investigate, whether prefork even has the problem, and if so, whether the core part of Jeff's patch already solves that. Patch: http://people.apache.org/~rjung/patches/keep-alive-while-exiting-trunk-20100601a.patch Before I proceed working on it, I would like some feedback, if this makes sense. It is a change in one of the more delicate parts and I think the patch is good enough now, to already review it. Some notes: 1) Implementation The worker threads keep running when the shutdown is graceful. The listener marks the process as quiescing, removes its sockets from the pollset and shuts them down. Then it proceeds looping until GracefulShutdownTimeout seconds are over. The timeout is necessary, because the pollset doesn't have an API to check, whether it is empty. So we don't know, whether there are still keep-alive connections. 2) Test You can set MaxRequestsPerChild to test this, or also test against the MaxSpareThreads by choosing a small difference between minSpare and MaxSpare. When using ab, a server-aborted connection will result in the following failure type: Failed requests:N (Connect: 0, Receive: 0, Length: N, Exceptions: 0) where N is the number of occurances. Although MaxrequestsPerChild is a convenient test case, keep in mind that for the worker and event MPMs we count keep-alive connections not requests. So you have to choose an appropriately low MaxRequestsPerChild in order to trigger it, especially when testing with ab -k which give you 100 requests per connection (in the default httpd configuration). 3) Configuration The patch uses the same KeepAliveWhileExiting witch, that is already part of Jeff's patch. The switch is a per VHost setting. For most of the above purposes it is used in a global context, because the MPM code often has no vhost in its hands (mostly no connection). To test, you have to set KeepAliveWhileExiting On globally and in the VHosts. For a final patch we would need to decide, whether KeepAliveWhileExiting becomes global only, or the global part of the patch gets another directive, or the feature is unconditionally switched on. The other relevant configuration setting is GracefulShutdownTimeout, until which time the graceful process shutdown is forced into an ungraceful one. 4) Related problems The following BZs seem to be related, but I didn't yet check the influence of the patch on them: https://issues.apache.org/bugzilla/show_bug.cgi?id=43081 https://issues.apache.org/bugzilla/show_bug.cgi?id=43359 5) Open questions - Should the behaviour be default? If not, should there be one global KeepAliveWhileExiting? - Any way of getting the size of a pollset? Or is using GracefulShutdownTimeout as a workaround OK? - there's possibly stuff which can be removed from the new remove_listeners_from_pollset() - Should there be a more fine-grained MPM state than AP_MPMQ_STOPPING, like AP_MPMQ_GRACEFUL and AP_MPMQ_STOPPING? This would make the patch a little nicer, especially the parts outside the MPM. - Anything to add for prefork? - Other MPMs? I didn't yet investigate the Windows MPM, but since there is only one child, I assume this kind of gracefulness doesn't make much sense? Any comments welcome. Regards, Rainer
Re: Fast by default
- Graham Leggett minf...@sharp.fm wrote: The very definition of tuned means tailored for your local setup. It's actually quite hard to get this thought accross. I think we should put it in a reboot of the performance ``optimization'' documentation. The default httpd configuration works reasonably well out the box. It is only when your site has special needs that it should start changing the setup, and the site should understand what their needs are and whether it is appropriate to turn it on. Zooming into mod_deflate, mod_deflate only makes sense if you have the CPU to support it. If you don't have enough CPU support (think virtualised hosts), mod_deflate will be a performance drag, not a boost. Typically, you would want to front a mod_deflate with an HTTP cache, such as mod_cache (or equivalent). Here mod_cache only makes sense if you have the disk space to support it, and there is no real one-size-fits-all cache setup. We arrive at one of my pet-peeve topic: Sane defaults. mod_cache's default values for CacheDirLevels and CacheDirLength will tear your file-system apart. And that is pretty much what: http://httpd.apache.org/docs/2.2/caching.html says... The next problem is that you only want to enable mod_deflate on compressible content - that means not images for most people, but might not be. Again, not every site has the same content, and therefore not every site has the same setup for mod_delate. This said, our default config is 15 years old, and attempts to disable deflate for browsers that don't support it, like Netscape 4. Unless there are modern browsers that have broken protocol support for transfer encoding, these obsolete examples need to be removed. An entirely different topic, but related to your proposal is that mod_deflate suggests to use a mod_filter based setup, but there's no example on how to configure that. (See my attached patch. Then you might understand why ;) Regards, Graham -- Bye, -- Igor Galić Tel: +43 (0) 699 122 96 338 Fax: +43(0) 1 91 333 41 Mail: i.ga...@brainsware.org URL: http://brainsware.org/ Index: mod/mod_deflate.xml === --- mod/mod_deflate.xml (revision 949964) +++ mod/mod_deflate.xml (working copy) @@ -44,32 +44,20 @@ AddOutputFilterByType DEFLATE text/html text/plain text/xml /example -pThe following configuration, while resulting in more compressed content, +pThe following configuration, while being the recommended setup using mod_filter, +results in more compressed content than the above but is also much more complicated. Do not use this unless you fully understand all the configuration details./p exampletitleCompress everything except images/title lt;Location /gt;br / indent -# Insert filterbr / -SetOutputFilter DEFLATEbr / -br / -# Netscape 4.x has some problems...br / -BrowserMatch ^Mozilla/4 gzip-only-text/htmlbr / -br / -# Netscape 4.06-4.08 have some more problemsbr / -BrowserMatch ^Mozilla/4\.0[678] no-gzipbr / -br / -# MSIE masquerades as Netscape, but it is finebr / -BrowserMatch \bMSIE !no-gzip !gzip-only-text/htmlbr / -# Don't compress imagesbr / -SetEnvIfNoCase Request_URI \br / -indent - \.(?:gif|jpe?g|png)$ no-gzip dont-varybr / -/indent -br / -# Make sure proxies don't deliver the wrong contentbr / -Header append Vary User-Agent env=!dont-varybr / + # Declare filter, add mod_deflate for all text/* MIME Typesbr / + FilterDeclare COMPRESSbr / + FilterProvider COMPRESS DEFLATE resp=Content-Type $text/br / + FilterChain COMPRESSbr / + FilterProtocol COMPRESS change=yes;byteranges=no + Header append Vary User-Agent env=!dont-varybr / /indent lt;/Locationgt; /example
Re: Fast by default
Typically, you would want to front a mod_deflate with an HTTP cache, such as mod_cache (or equivalent). Here mod_cache only makes sense if you have the disk space to support it, and there is no real one-size-fits-all cache setup. This said, our default config is 15 years old, and attempts to disable deflate for browsers that don't support it, like Netscape 4. Unless there are modern browsers that have broken protocol support for transfer encoding, these obsolete examples need to be removed. These go together a bit Vary, the current workarounds for old browsers cause Vary: User-Agent which is quite a buzzkill for a cache! -- Eric Covener cove...@gmail.com
Re: Fast by default
Geez, Eric. No wonder people don't want to contribute to httpd, when they run into an attitude like yours. That dismissiveness makes me embarressed for our community. There is zero reason for us to avoid putting deflate into the default configuration. It is also very arguable that we should leave it off. I think others have argued well to enable it by default, while you've simply dismissed them with your holier-than-thou attitude and lack of any solid rationale. -g On May 31, 2010 8:06 PM, Eric Covener cove...@gmail.com wrote: On Mon, May 31, 2010 at 8:30 PM, Bryan McQuade bmcqu...@google.com wrote: I propose providing an... An additional httpd.conf doesn't sound valuable to me. What slice of non-savvy users would scrutinize an alternate config file, can replace the config file of their webserver, isn't using a webserver packaged by their OS, and wouldn't have just gotten the same information today from the manual and 400,000 other websites? There's currently no ifModule bloat in the default conf, but you're welcome to submit a patch that adds one for deflate or expires (latter seems more unwise to me). See the supplemental configuration section of the generated config. This doesn't address mass-vhost companies failing to allow deflate because it's not in the no-args HTTPD ./configure , which sounds far-fetched to me. I can't recall a users@ or #httpd user implying being subjected to such a thing with their own build or with cheap hosting. -- Eric Covener cove...@gmail.com
RE: Fast by default
From: Greg SteinSent: Dienstag, 1. Juni 2010 14:40 To: dev@httpd.apache.org Subject: Re: Fast by default Geez, Eric. No wonder people don't want to contribute to httpd, when they run into an attitude like yours. That dismissiveness makes me embarressed for our community. There is zero reason for us to avoid putting deflate into the default configuration. It is also very arguable that we should leave it off. I think others have argued well to enable it by default, while you've simply dismissed them with your holier-than-thou attitude and lack of any solid rationale. And others have argued well to leave it off by default. My personal opinion is that we should leave it disabled by default for the reasons (CPU, Proxies, Browser behaviour, ETAG problem) mentioned by others. Regards Rüdiger -g On May 31, 2010 8:06 PM, Eric Covener cove...@gmail.com wrote: On Mon, May 31, 2010 at 8:30 PM, Bryan McQuade bmcqu...@google.com wrote: I propose providing an... An additional httpd.conf doesn't sound valuable to me. What slice of non-savvy users would scrutinize an alternate config file, can replace the config file of their webserver, isn't using a webserver packaged by their OS, and wouldn't have just gotten the same information today from the manual and 400,000 other websites? There's currently no ifModule bloat in the default conf, but you're welcome to submit a patch that adds one for deflate or expires (latter seems more unwise to me). See the supplemental configuration section of the generated config. This doesn't address mass-vhost companies failing to allow deflate because it's not in the no-args HTTPD ./configure , which sounds far-fetched to me. I can't recall a users@ or #httpd user implying being subjected to such a thing with their own build or with cheap hosting. -- Eric Covener cove...@gmail.com
Enhanced error log format for trunk?
I repeatedly inserted millisecond or microsecond timestamps as well as
PID and thread ID information into the ErrorLog when trying to diagnose
problems, most often in combination with additional log lines.
Due to the increased load and capability of systems and increasing
amount of concurrency, would those features be interesting by default?
1) Sub-second timestamps in error log
Index: server/util_time.c
===
--- server/util_time.c (revision 949733)
+++ server/util_time.c (working copy)
@@ -151,6 +151,8 @@
apr_time_exp_t xt;
const char *s;
int real_year;
+int usec;
+int div;
/* example: Wed Jun 30 21:49:08 1993 */
/* 123456789012345678901234 */
@@ -177,6 +179,12 @@
*date_str++ = ':';
*date_str++ = xt.tm_sec / 10 + '0';
*date_str++ = xt.tm_sec % 10 + '0';
+*date_str++ = '.';
+usec = (int)xt.tm_usec;
+for (div=10; div0; div=div/10) {
+*date_str++ = usec / div + '0';
+usec = usec % div;
+}
*date_str++ = ' ';
real_year = 1900 + xt.tm_year;
*date_str++ = real_year / 1000 + '0';
Index: server/log.c
===
--- server/log.c(revision 949733)
+++ server/log.c(working copy)
@@ -594,9 +594,9 @@
if (logf ((level APLOG_STARTUP) != APLOG_STARTUP)) {
errstr[0] = '[';
ap_recent_ctime(errstr + 1, apr_time_now());
-errstr[1 + APR_CTIME_LEN - 1] = ']';
-errstr[1 + APR_CTIME_LEN] = ' ';
-len = 1 + APR_CTIME_LEN + 1;
+errstr[1 + APR_CTIME_LEN + 7 - 1] = ']';
+errstr[1 + APR_CTIME_LEN + 7] = ' ';
+len = 1 + APR_CTIME_LEN + 7 + 1;
} else {
len = 0;
}
and maybe switching away from the CTIME name, because it's no longer
real ctime.
2) PID and thread ID
Index: server/log.c
===
--- server/log.c(revision 949733)
+++ server/log.c(working copy)
@@ -25,6 +25,7 @@
#include apr_general.h/* for signal stuff */
#include apr_strings.h
#include apr_errno.h
+#include apr_portable.h
#include apr_thread_proc.h
#include apr_lib.h
#include apr_signal.h
@@ -606,6 +607,18 @@
[%s] , priorities[level_and_mask].t_name);
}
+len += apr_snprintf(errstr + len, MAX_STRING_LEN - len,
+[% APR_PID_T_FMT, getpid());
+#if APR_HAS_THREADS
+{
+apr_os_thread_t tid = apr_os_thread_current();
+len += apr_snprintf(errstr + len, MAX_STRING_LEN - len,
+:%pT, tid);
+}
+#endif
+errstr[len++] = ']';
+errstr[len++] = ' ';
+
if (file level_and_mask == APLOG_DEBUG) {
#if defined(_OSD_POSIX) || defined(WIN32) || defined(__MVS__)
char tmp[256];
PID and thread id are useful, because the can be correlated with the
access_log (when added there). they also help to understand, which
messages belong together when multiple log events are intertwined.
3) Related things for access log
There's a related open Bugzilla about adding sub-second timestamps (not:
duration) to the access log (BZ 49251). The OP suggests to use %M for an
absolute timestamp. It would be nice to improve the strftime use of
%{format}t to allow for a proprietary microsecond letter which will add
the microsecond part of the actual time. I found no standards defined
for that.
4) General correlation improvements
To be able to correlate error and access log, it would be helpful to
share a common id, e.g. the unique_id, and be able to log it in both
files. The id generated by mod_unique_id comes too late though
(post_read_request). Since it actually only uses the request timestamp
and the connection id of the request, it could be calculated much earlier.
Regards,
Rainer
mod_disk_cache: recommended defaults
Hi folks, here's a patch to mod_disk_cache.h to set the defaults as recommended by: http://httpd.apache.org/docs/trunk/mod/mod_disk_cache.html This is quite a complex patch, and I'm not sure it'll pass any reviews. Index: mod_disk_cache.h === --- mod_disk_cache.h(revision 949964) +++ mod_disk_cache.h(working copy) @@ -79,7 +79,7 @@ /* TODO: Make defaults OS specific */ #define CACHEFILE_LEN 20/* must be less than HASH_LEN/2 */ #define DEFAULT_DIRLEVELS 2 -#define DEFAULT_DIRLENGTH 2 +#define DEFAULT_DIRLENGTH 1 #define DEFAULT_MIN_FILE_SIZE 1 #define DEFAULT_MAX_FILE_SIZE 100 -- Igor Galić Tel: +43 (0) 699 122 96 338 Fax: +43(0) 1 91 333 41 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
Re: mod_disk_cache: recommended defaults
- Eric Covener cove...@gmail.com wrote: here's a patch to mod_disk_cache.h to set the defaults as recommended by: http://httpd.apache.org/docs/trunk/mod/mod_disk_cache.html I might be crazy, but I can't spot the recommendation you refer to in the doc. No, you're not crazy. I supplied the wrong link: http://httpd.apache.org/docs/trunk/caching.html#disk ``Unless you have a good reason not to, using a setting of 1 for CacheDirLength is recommended.'' -- Eric Covener cove...@gmail.com Bye, -- Igor Galić Tel: +43 (0) 699 122 96 338 Fax: +43(0) 1 91 333 41 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
mod_deflate handling of empty initial brigade
In a filter module I'm writing, it's possible for the first pass
through the output filter to end up calling:
ap_pass_brigade(f-next, an_empty_brigade)
If mod_deflate's output filter appears later in the output filter
chain, bad things happen:
1. On the first trip through the output filter chain, mod_deflate just
passes the empty brigade through to the next filter:
/* Do nothing if asked to filter nothing. */
if (APR_BRIGADE_EMPTY(bb)) {
return ap_pass_brigade(f-next, bb);
}
2. Control eventually reaches core_output_filter, which sends the
response headers.
3. On a subsequent trip through the output filter chain, my module calls:
ap_pass_brigade(f-next, a non_empty_brigade)
4. Upon seeing the non empty brigade, mod_deflate's output filter does
all the initialization that it would normally do at the start of a
response. If the response is compressible, deflate_out_filter adds
Content-Encoding: gzip to the output headers. The output headers
have been sent already, though, so the Content-Encoding never gets
sent.
5. The client receives a compressed response without a Content-Encoding header.
I can prevent this by not calling ap_pass_brigade in my module until I
have the first non-empty brigade for the rest of the output filter
chain to work with.
I'm curious, though: which module is doing the wrong thing:
- My module, by sending an empty brigade through the rest of the
output filter chain prior to sending the first non-empty brigade?
- Or mod_deflate, by adding fields to the response header after
calling ap_pass_brigade?
Thanks,
-Brian
Re: mod_deflate handling of empty initial brigade
On Tue, Jun 1, 2010 at 10:39 AM, Brian Pane brianp...@gmail.com wrote:
In a filter module I'm writing, it's possible for the first pass
through the output filter to end up calling:
ap_pass_brigade(f-next, an_empty_brigade)
If mod_deflate's output filter appears later in the output filter
chain, bad things happen:
1. On the first trip through the output filter chain, mod_deflate just
passes the empty brigade through to the next filter:
/* Do nothing if asked to filter nothing. */
if (APR_BRIGADE_EMPTY(bb)) {
return ap_pass_brigade(f-next, bb);
}
2. Control eventually reaches core_output_filter, which sends the
response headers.
3. On a subsequent trip through the output filter chain, my module calls:
ap_pass_brigade(f-next, a non_empty_brigade)
4. Upon seeing the non empty brigade, mod_deflate's output filter does
all the initialization that it would normally do at the start of a
response. If the response is compressible, deflate_out_filter adds
Content-Encoding: gzip to the output headers. The output headers
have been sent already, though, so the Content-Encoding never gets
sent.
5. The client receives a compressed response without a Content-Encoding
header.
I can prevent this by not calling ap_pass_brigade in my module until I
have the first non-empty brigade for the rest of the output filter
chain to work with.
I'm curious, though: which module is doing the wrong thing:
- My module, by sending an empty brigade through the rest of the
output filter chain prior to sending the first non-empty brigade?
- Or mod_deflate, by adding fields to the response header after
calling ap_pass_brigade?
Would it do harm if the core output filter did the same check as
mod_deflate before committing the headers?
Seems like a filter could have just dropped the content of your first
brigade anyway.
--
Eric Covener
cove...@gmail.com
Re: mod_deflate handling of empty initial brigade
The guide to writing output filters says:
https://httpd.apache.org/docs/trunk/developer/output-filters.html#invocation
An output filter should never pass an empty brigade down the filter
chain. But, for good defensive programming, filters should be prepared
to accept an empty brigade, and do nothing.
According to this, mod_deflate should not pass the empty brigade
along, and your module also should not be passing and empty brigade to
mod_deflate.
Eric, others, should we submit a patch to mod_deflate to change its
behavior to be consistent with the output filter guide?
On Tue, Jun 1, 2010 at 10:39 AM, Brian Pane brianp...@gmail.com wrote:
In a filter module I'm writing, it's possible for the first pass
through the output filter to end up calling:
ap_pass_brigade(f-next, an_empty_brigade)
If mod_deflate's output filter appears later in the output filter
chain, bad things happen:
1. On the first trip through the output filter chain, mod_deflate just
passes the empty brigade through to the next filter:
/* Do nothing if asked to filter nothing. */
if (APR_BRIGADE_EMPTY(bb)) {
return ap_pass_brigade(f-next, bb);
}
2. Control eventually reaches core_output_filter, which sends the
response headers.
3. On a subsequent trip through the output filter chain, my module calls:
ap_pass_brigade(f-next, a non_empty_brigade)
4. Upon seeing the non empty brigade, mod_deflate's output filter does
all the initialization that it would normally do at the start of a
response. If the response is compressible, deflate_out_filter adds
Content-Encoding: gzip to the output headers. The output headers
have been sent already, though, so the Content-Encoding never gets
sent.
5. The client receives a compressed response without a Content-Encoding
header.
I can prevent this by not calling ap_pass_brigade in my module until I
have the first non-empty brigade for the rest of the output filter
chain to work with.
I'm curious, though: which module is doing the wrong thing:
- My module, by sending an empty brigade through the rest of the
output filter chain prior to sending the first non-empty brigade?
- Or mod_deflate, by adding fields to the response header after
calling ap_pass_brigade?
Thanks,
-Brian
canned deflate conf in manual -- time to drop the NS4/vary?
IIUC, the vary: user-agent to accomodate Netscape 4 is a pain for caches because obviously they can only vary on the entire user-agent. http://httpd.apache.org/docs/2.2/mod/mod_deflate.html Is it time to move this aspect of the snippet into a separate note or some historical trivia section, to remove the Vary? -- On the same topic, are there still non-academic CSS and JS compression issues (e.g. XP-era browsers, earlier, later, ???) Should we instead account for these in the complicated/more compression example, and is there a way to do so without adding the Vary right back in? -- Eric Covener cove...@gmail.com
RE: mod_deflate handling of empty initial brigade
-Original Message- From: Bryan McQuade Sent: Dienstag, 1. Juni 2010 16:54 To: dev@httpd.apache.org Cc: mdste...@google.com Subject: Re: mod_deflate handling of empty initial brigade The guide to writing output filters says: https://httpd.apache.org/docs/trunk/developer/output-filters.h tml#invocation An output filter should never pass an empty brigade down the filter chain. But, for good defensive programming, filters should be prepared to accept an empty brigade, and do nothing. IMHO do nothing means the same thing as pass it down the chain. So I think mod_deflate does the correct thing here. Regards Rüdiger
Re: mod_deflate handling of empty initial brigade
On Tue, Jun 1, 2010 at 11:02 AM, Plüm, Rüdiger, VF-Group
ruediger.pl...@vodafone.com wrote:
The guide to writing output filters says:
https://httpd.apache.org/docs/trunk/developer/output-filters.h
tml#invocation
An output filter should never pass an empty brigade down the filter
chain. But, for good defensive programming, filters should be prepared
to accept an empty brigade, and do nothing.
IMHO do nothing means the same thing as pass it down the chain. So I think
mod_deflate does the correct thing here.
On the contrary, the link above specifically recommends putting the
following code snippet at the top of your output filter to suppress
the empty brigade (rather than pass it along to the next filter):
apr_status_t dummy_filter(ap_filter_t *f, apr_bucket_brigade *bb)
{
if (APR_BRIGADE_EMPTY(bb)) {
return APR_SUCCESS;
}
Does anyone know whether or not this is the right thing to do?
Cheers,
-Matt
Re: RFC: drop support for OpenSSL 1.0 in trunk/2.3?
Deprecating obsolete libraries is a good thing, especially if there is a compelling replacement. I think this goes hand in hand with what operating system versions we will be targeting for 2.4. We should inventory which versions of the libraries are offered on each and then make the decision whether to accomodate: * Windows: none * Mac OS X 10.6: OpenSSL 0.9.8l 5 Nov 2009 * FreeBSD 6.4-STABLE: OpenSSL 0.9.7e-p1 25 Oct 2004 * FreeBSD 7.2-STABLE: OpenSSL 0.9.8e 23 Feb 2007 * FreeBSD 8-STABLE: OpenSSL 0.9.8k 25 Mar 2009 * OpenBSD 4.6: OpenSSL 0.9.8k 25 Mar 2009 * Solaris 10: 0.9.7 with backports... don't recall what's in the Coolstack but someone else may be able to tell us. The Coolstack and the Webstack both use the system's SSL bindings. Coolstack symlinks it: libssl.so.0.9.7 = /usr/sfw/lib/libssl.so.0.9.7 lrwxrwxrwx 1 root root 28 Jul 13 2009 /opt/coolstack/lib/libssl.so.0.9.7 - /usr/sfw/lib/libssl.so.0.9.7 while webstack links it directly: libcrypto.so.0.9.7 = /usr/sfw/lib/libcrypto.so.0.9.7 Bye, -- Igor Galić Tel: +43 (0) 699 122 96 338 Fax: +43(0) 1 91 333 41 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
Re: mod_deflate handling of empty initial brigade
On 1 Jun 2010, at 15:53, Bryan McQuade wrote: According to this, mod_deflate should not pass the empty brigade along, and your module also should not be passing and empty brigade to mod_deflate. Eric, others, should we submit a patch to mod_deflate to change its behavior to be consistent with the output filter guide? +1. Looks like a trivial fix if this is the only error path concerned. And an extra-robustness check in core output filter can't hurt (can it?) -- Nick Kew
Re: Fast by default
On 6/1/2010 7:05 AM, Eric Covener wrote: Typically, you would want to front a mod_deflate with an HTTP cache, such as mod_cache (or equivalent). Here mod_cache only makes sense if you have the disk space to support it, and there is no real one-size-fits-all cache setup. This said, our default config is 15 years old, and attempts to disable deflate for browsers that don't support it, like Netscape 4. Unless there are modern browsers that have broken protocol support for transfer encoding, these obsolete examples need to be removed. These go together a bit Vary, the current workarounds for old browsers cause Vary: User-Agent which is quite a buzzkill for a cache! Both is out of the question, for the reason you mention. I would argue for neither cache nor deflate, but for a bit different reason. But I see no reason to leave these out of modules 'most', to be ready for the user to deploy them. Cache should be disabled by default simply because it confuses beginning users, and we can't expect they will read all the relevant docs about setting proper expiries on their content. They will undoubtedly be lost, just as we've all been lost reconfiguring our servers and not realizing the 'refresh' wasn't a full refresh on IE, FF or what have you. It will generate too many invalid bug reports and user list queries. The current user list traffic around the cache module seems quite reasonable. Deflate should be disabled simply because the user can't trivially inspect the stream in forensics, and it interacts in odd ways once cache is enabled. I like to think of a fresh httpd install as something the common user can wrap their head around, and we even get user confusion around chunking. Plus deflate may provide no benefit, and degrade performance, if the CPU utilization is a greater concern than bandwidth utilization.
What's next for 2.2 and 2.3/trunk?
Considering that 2.3/trunk is back to limbo-land, I'd like to propose that we be more aggressive is backporting some items. Even if under experimental, it would be nice if slotmem and socache were backported. I also like the refactoring of the providers for proxy in trunk as compared to 2.2, but last time I suggested it, it looked like 2.3/2.4 was close(r) to reality... comments...?
Re: drop support for OpenSSL 1.0 in trunk/2.3?
On 25.05.2010 15:09, Plüm, Rüdiger, VF-Group wrote: -Original Message- From: Joe Orton Sent: Dienstag, 25. Mai 2010 14:46 To: dev@httpd.apache.org Subject: RFC: drop support for OpenSSL 1.0 in trunk/2.3? I'd like to drop support for versions of OpenSSL older than 1.0 in the trunk mod_ssl. We have 200+ lines of compat macro junk and still six different compiler warnings remain in a trunk build against 1.0.0. pro: simplify code: remove ssl_toolkit_compat.h and all compat macro mess which litters the code pro: simplify testing: no longer have to test/worry about regressing builds against N subtly different versions of the OpenSSL API all pro: can drop the internal CRL revocation code in favour of OpenSSL's pro: users will be encouraged to upgrade to a modern OpenSSL which has secure TLS reneg con: trunk/2.3 won't build on all platforms/distros which ship natively with OpenSSL 1.0 (duh) While the pros sound promising this is a real strong con. Especially as this would mean that 2.4 would not work with OpenSSL 1.0. The problem I see is that if you want to use other OS provided libraries like openldap they have dependencies on the OS provided OpenSSL and binding Apache against a different OpenSSL version as these libraries are bound against looks like a big problem if Apache is bound to them as well. And building a whole stack of dependencies for Apache seems to be a too large hurdle for me for adoption. So currently I would be -1 (vote not veto) on this. The same for me. Supporting only 0.9.8 and newer seems to be OK w.r.t. to supported platforms and what they provide now or what can be expected from them. Deciding about a minimum 0.9.8 patch version is harder. Although it would be good if vendors would support secure reneg soon, I doubt that most users will have it on their servers in the next few years. Some might get a backport into the vendor supplied version, but not really a full 0.9.8n or higher. So I'd be +1 for dropping support for OpenSSL 0.9.8. Regards, Rainer
Re: mod_deflate handling of empty initial brigade
I went ahead and created a bug entry/patch to make the (trivial) change to mod_deflate to make it conform to the Guide to writing output filters: https://issues.apache.org/bugzilla/show_bug.cgi?id=49369 Brian, does this patch to mod_deflate fix your problem? Nick, does this change seem reasonable to you? (I tried to follow the directions for patches on http://httpd.apache.org/dev/patches.html; apologies if I got it wrong...) Cheers, -Matt On Tue, Jun 1, 2010 at 11:54 AM, Nick Kew n...@webthing.com wrote: On 1 Jun 2010, at 15:53, Bryan McQuade wrote: According to this, mod_deflate should not pass the empty brigade along, and your module also should not be passing and empty brigade to mod_deflate. Eric, others, should we submit a patch to mod_deflate to change its behavior to be consistent with the output filter guide? +1. Looks like a trivial fix if this is the only error path concerned. And an extra-robustness check in core output filter can't hurt (can it?) -- Nick Kew
Re: canned deflate conf in manual -- time to drop the NS4/vary?
Yeah, it should only Vary on Accept-encoding (already does). It's still not perfect, but at least it doesn't blow up proxies too much. The question to people with statistics - are there any other issues with gzip/proxy configurations? Sergey On Tue, Jun 1, 2010 at 11:01 AM, Eric Covener cove...@gmail.com wrote: IIUC, the vary: user-agent to accomodate Netscape 4 is a pain for caches because obviously they can only vary on the entire user-agent. http://httpd.apache.org/docs/2.2/mod/mod_deflate.html Is it time to move this aspect of the snippet into a separate note or some historical trivia section, to remove the Vary? -- On the same topic, are there still non-academic CSS and JS compression issues (e.g. XP-era browsers, earlier, later, ???) Should we instead account for these in the complicated/more compression example, and is there a way to do so without adding the Vary right back in? -- Eric Covener cove...@gmail.com
Re: What's next for 2.2 and 2.3/trunk?
On Tue, Jun 1, 2010 at 9:08 AM, Jim Jagielski j...@apache.org wrote: Considering that 2.3/trunk is back to limbo-land, I'd like to propose that we be more aggressive is backporting some items. Even if under experimental, it would be nice if slotmem and socache were backported. I also like the refactoring of the providers for proxy in trunk as compared to 2.2, but last time I suggested it, it looked like 2.3/2.4 was close(r) to reality... comments...? 2.2 came out 5.5 years ago. I strongly support shipping 2.4. i've not had time to try to ship more alpha/betas. Does anyone else have time to head up RMing it? Its easy once you get it down, 2 commits, one run of of the release shell script, and 3 emails.
Re: canned deflate conf in manual -- time to drop the NS4/vary?
Don't forget the ongoing issue that if you ONLY vary on 'Accept-Encoding' then almost ALL browsers will then refuse to cache a response entity LOCALLY and the pain factor moves directly to the Proxy/Content Server(s). If you vary on 'User-Agent' ( No longer reasonable because of the abuse of that header 'out there'? ) then the browsers WILL cache responses locally and the pain is reduced at the Proxy/Content server level, but pie is not free at a truck stop and there are then OTHER issues to deal with. The OTHER 'ongoing issue' regarding compression is that, to this day, it still ONLY works for a limited set of MIME types. The 'Accept-Encoding: gzip,deflate' header coming from ALL major browser is still mostly a LIE. It would seem to indicate that the MIME type doesn't matter and it will 'decode' for ANY MIME type but nothing could be further from the truth. There is no browser on the planet that will 'Accept-Encoding' for ANY/ALL mime type(s). If you are going to turn compression ON by default, without the user having to make any decisions for their particular environment, then part of the decision for the default config has to be 'Which MIME types?' text/plain and/or text/html only? SOME browsers can 'Accept-Encoding' on the ever-increasing .js Javascript backloads but some CANNOT. These 2 issues alone are probably enough to justify keeping compression OFF by default. A lot of people that use Apache won't even be able to get their heads around either one of these 'issues' and they really SHOULD do a little homework before turning it ON. Someone already quoted that... 'people expect the default config to just WORK without major issues'. That's exactly what you have now. It's not 'broken'. Why change it? Kevin Kiley -Original Message- From: Sergey Chernyshev sergey.chernys...@gmail.com To: dev@httpd.apache.org Sent: Tue, Jun 1, 2010 3:09 pm Subject: Re: canned deflate conf in manual -- time to drop the NS4/vary? Yeah, it should only Vary on Accept-encoding (already does). It's still not perfect, but at least it doesn't blow up proxies too much. The question to people with statistics - are there any other issues with gzip/proxy configurations? Sergey On Tue, Jun 1, 2010 at 11:01 AM, Eric Covener cove...@gmail.com wrote: IIUC, the vary: user-agent to accomodate Netscape 4 is a pain for caches because obviously they can only vary on the entire user-agent. http://httpd.apache.org/docs/2.2/mod/mod_deflate.html Is it time to move this aspect of the snippet into a separate note or some historical trivia section, to remove the Vary? -- On the same topic, are there still non-academic CSS and JS compression issues (e.g. XP-era browsers, earlier, later, ???) Should we instead account for these in the complicated/more compression example, and is there a way to do so without adding the Vary right back in? -- Eric Covener cove...@gmail.com
Re: canned deflate conf in manual -- time to drop the NS4/vary?
On Tue, 01 Jun 2010 17:44:41 -0400 toki...@aol.com wrote: Don't forget the ongoing issue that if you ONLY vary on 'Accept-Encoding' then almost ALL browsers will then refuse to cache a response entity LOCALLY Really? That sounds bizarre! Do you have a reference for it? -- Nick Kew
Re: Fast by default
There is zero reason for us to avoid putting deflate into the default configuration. Sorry. There ARE (good) reasons to avoid doing so. I'm the one who wrote the FIRST mod_gzip module for Apache 1.x series so you would think I'd be a strong advocate of 'auto-enablement' by default, but I am NOT. There is HOMEWORK involved here and most users will get into deep tapioca unless they understand all the (ongoing) issues. it is also very arguable that we should leave it off. Yes, it is. I think others have argued well to enable it by default. Disagree. I haven't seen the 'good' argument for 'auto-enablement' yet. Some of the reasons to NOT 'go there' are coming out in other similar threads right now... Here's a clip from the (concurrent) message thread entitled... 'Canned deflate conf in manual - time to drop the NS4/Vary' [snip] Don't forget the ongoing issue that if you ONLY vary on 'Accept-Encoding' then almost ALL browsers will then refuse to cache a response entity LOCALLY and the pain factor moves directly to the Proxy/Content Server(s). If you vary on 'User-Agent' ( No longer reasonable because of the abuse of that header 'out there'? ) then the browsers WILL cache responses locally and the pain is reduced at the Proxy/Content server level, but pie is not free at a truck stop and there are then OTHER issues to deal with. The OTHER 'ongoing issue' regarding compression is that, to this day, it still ONLY works for a limited set of MIME types. The 'Accept-Encoding: gzip,deflate' header coming from ALL major browser is still mostly a LIE. It would seem to indicate that the MIME type doesn't matter and it will 'decode' for ANY MIME type but nothing could be further from the truth. There is no browser on the planet that will 'Accept-Encoding' for ANY/ALL mime type(s). If you are going to turn compression ON by default, without the user having to make any decisions for their particular environment, then part of the decision for the default config has to be 'Which MIME types?' text/plain and/or text/html only? SOME browsers can 'Accept-Encoding' on the ever-increasing .js Javascript backloads but some CANNOT. These 2 issues alone are probably enough to justify keeping compression OFF by default. A lot of people that use Apache won't even be able to get their heads around either one of these 'issues' and they really SHOULD do a little homework before turning it ON. Someone already quoted that... 'people expect the default config to just WORK without major issues'. That's exactly what you have now. It's not 'broken'. Why change it? Kevin Kiley [snip] -Original Message- From: Greg Stein gst...@gmail.com To: dev@httpd.apache.org Sent: Tue, Jun 1, 2010 7:40 am Subject: Re: Fast by default Geez, Eric. No wonder people don't want to contribute to httpd, when they run into an attitude like yours. That dismissiveness makes me embarressed for our community. There is zero reason for us to avoid putting deflate into the default configuration. It is also very arguable that we should leave it off. I think others have argued well to enable it by default, while you've simply dismissed them with your holier-than-thou attitude and lack of any solid rationale. -g On May 31, 2010 8:06 PM, Eric Covener cove...@gmail.com wrote: On Mon, May 31, 2010 at 8:30 PM, Bryan McQuade bmcqu...@google.com wrote: I propose providing an... An additional httpd.conf doesn't sound valuable to me. What slice of non-savvy users would scrutinize an alternate config file, can replace the config file of their webserver, isn't using a webserver packaged by their OS, and wouldn't have just gotten the same information today from the manual and 400,000 other websites? There's currently no ifModule bloat in the default conf, but you're welcome to submit a patch that adds one for deflate or expires (latter seems more unwise to me). See the supplemental configuration section of the generated config. This doesn't address mass-vhost companies failing to allow deflate because it's not in the no-args HTTPD ./configure , which sounds far-fetched to me. I can't recall a users@ or #httpd user implying being subjected to such a thing with their own build or with cheap hosting. -- Eric Covener cove...@gmail.com
Re: Fast by default
web sites are loading too slow for pipes and web-server power that we have. The key phrase there is 'that WE have'. YOU need to tune YOUR configs to match what YOU have. ANYONE who uses Apache can/should/must do that. That's how that works. The discussion at this moment is what 'default' configs should ship with Apache. It is NOT POSSIBLE to accomodate EVERYONE. The default httpd.conf for Apache is simply JFW ( Just Feckin Works )... and for a product as complicated as Apache I tend to agree with those who think that is all that it needs to 'ship' with. Kevin Kiley -Original Message- From: Sergey Chernyshev sergey.chernys...@gmail.com To: dev@httpd.apache.org Sent: Tue, Jun 1, 2010 5:30 pm Subject: Re: Fast by default It's not 'broken'. Why change it? Please don't think that old configurations and practices are not broken - web sites are loading too slow for pipes and web-server power that we have. And situation is getting worse year after year - here's analysis by Patrick Meanan of WebPageTest.org's one year history: http://blog.patrickmeenan.com/2010/05/are-pages-getting-faster.html Sergey Kevin Kiley [snip] -Original Message- From: Greg Stein gst...@gmail.com To: dev@httpd.apache.org Sent: Tue, Jun 1, 2010 7:40 am Subject: Re: Fast by default Geez, Eric. No wonder people don't want to contribute to httpd, when they run into an attitude like yours. That dismissiveness makes me embarressed for our community. There is zero reason for us to avoid putting deflate into the default configuration. It is also very arguable that we should leave it off. I think others have argued well to enable it by default, while you've simply dismissed them with your holier-than-thou attitude and lack of any solid rationale. -g On May 31, 2010 8:06 PM, Eric Covener cove...@gmail.com wrote: On Mon, May 31, 2010 at 8:30 PM, Bryan McQuade bmcqu...@google.com wrote: I propose providing an... An additional httpd.conf doesn't sound valuable to me. What slice of non-savvy users would scrutinize an alternate config file, can replace the config file of their webserver, isn't using a webserver packaged by their OS, and wouldn't have just gotten the same information today from the manual and 400,000 other websites? There's currently no ifModule bloat in the default conf, but you're welcome to submit a patch that adds one for deflate or expires (latter seems more unwise to me). See the supplemental configuration section of the generated config. This doesn't address mass-vhost companies failing to allow deflate because it's not in the no-args HTTPD ./configure , which sounds far-fetched to me. I can't recall a users@ or #httpd user implying being subjected to such a thing with their own build or with cheap hosting. -- Eric Covener cove...@gmail.com
Re: canned deflate conf in manual -- time to drop the NS4/vary?
Sergey wrote... That's new to me that browsers don't cache stuff that has Vary only on Accept-Encoding - can you post some statistics or describe the test you ran? Test results and statistics... Apache DEV forum... http://www.pubbs.net/200908/httpd/55434-modcache-moddeflate-and-vary-user-agent.html apache-modgzip forum... http://marc.info/?l=apache-modgzipm=103958533520502w=2 Etc, etc. Lots of discussion about this has taken place over on the SQUID forums as well. As for *all* content types, I don't think we're talking about compressing images and it's relatively easy to create a white-list to have gzip on for by default. Apache's own mod_deflate docs show how to exclude images. That's a no-brainer. It's the OTHER mime types that get hairy. The question regarding support in browsers actually is very serious too and I'd love to see statistics for that too - it sounds too scary and middle-ages to me. You must be new to this sort of thing. See links above and read the MANY related threads on the SQUID forum. I didn't get this impression from all the talks about gzip and research that guys from Google did, for example, when they were looking for a source of lower gzip rates (it turned out to be antivirus software stripping Accept-encoding headers). I think I know the Google R/D you are referring to and it was almost a joke. There was a LOT of research they did NOT do and they made many assumptions that are simply NOT TRUE in the REAL WORLD. Thank you, Sergey You're welcome Kevin
Re: Fast by default
On Tue, Jun 1, 2010 at 16:25, Sergey Chernyshev sergey.chernys...@gmail.com wrote: This sounds scary! How do large companies enable gzip then? How many hoops do they jump through? sounds like those hoops are in thousands! And I don't understand how one company's setup would be different from another still, even if situation is that bad as you describe it. What kind of trade-offs do large companies go for when they enable gzip? more overall traffic? no cache? Sergey On Tue, Jun 1, 2010 at 6:17 PM, toki...@aol.com wrote: There is zero reason for us to avoid putting deflate into the default configuration. Sorry. There ARE (good) reasons to avoid doing so. I'm the one who wrote the FIRST mod_gzip module for Apache 1.x series so you would think I'd be a strong advocate of 'auto-enablement' by default, but I am NOT. There is HOMEWORK involved here and most users will get into deep tapioca unless they understand all the (ongoing) issues. it is also very arguable that we should leave it off. Yes, it is. I think others have argued well to enable it by default. Disagree. I haven't seen the 'good' argument for 'auto-enablement' yet. Some of the reasons to NOT 'go there' are coming out in other similar threads right now... Here's a clip from the (concurrent) message thread entitled... 'Canned deflate conf in manual - time to drop the NS4/Vary' [snip] Don't forget the ongoing issue that if you ONLY vary on 'Accept-Encoding' then almost ALL browsers will then refuse to cache a response entity LOCALLY and the pain factor moves directly to the Proxy/Content Server(s). If you vary on 'User-Agent' ( No longer reasonable because of the abuse of that header 'out there'? ) then the browsers WILL cache responses locally and the pain is reduced at the Proxy/Content server level, but pie is not free at a truck stop and there are then OTHER issues to deal with. The OTHER 'ongoing issue' regarding compression is that, to this day, it still ONLY works for a limited set of MIME types. The 'Accept-Encoding: gzip,deflate' header coming from ALL major browser is still mostly a LIE. It would seem to indicate that the MIME type doesn't matter and it will 'decode' for ANY MIME type but nothing could be further from the truth. There is no browser on the planet that will 'Accept-Encoding' for ANY/ALL mime type(s). If you are going to turn compression ON by default, without the user having to make any decisions for their particular environment, then part of the decision for the default config has to be 'Which MIME types?' text/plain and/or text/html only? SOME browsers can 'Accept-Encoding' on the ever-increasing .js Javascript backloads but some CANNOT. These 2 issues alone are probably enough to justify keeping compression OFF by default. A lot of people that use Apache won't even be able to get their heads around either one of these 'issues' and they really SHOULD do a little homework before turning it ON. Someone already quoted that... 'people expect the default config to just WORK without major issues'. That's exactly what you have now. It's not 'broken'. Why change it? Kevin Kiley [snip] -Original Message- From: Greg Stein gst...@gmail.com To: dev@httpd.apache.org Sent: Tue, Jun 1, 2010 7:40 am Subject: Re: Fast by default Geez, Eric. No wonder people don't want to contribute to httpd, when they run into an attitude like yours. That dismissiveness makes me embarressed for our community. There is zero reason for us to avoid putting deflate into the default configuration. It is also very arguable that we should leave it off. I think others have argued well to enable it by default, while you've simply dismissed them with your holier-than-thou attitude and lack of any solid rationale. -g On May 31, 2010 8:06 PM, Eric Covener cove...@gmail.com wrote: On Mon, May 31, 2010 at 8:30 PM, Bryan McQuade bmcqu...@google.com wrote: I propose providing an... An additional httpd.conf doesn't sound valuable to me. What slice of non-savvy users would scrutinize an alternate config file, can replace the config file of their webserver, isn't using a webserver packaged by their OS, and wouldn't have just gotten the same information today from the manual and 400,000 other websites? There's currently no ifModule bloat in the default conf, but you're welcome to submit a patch that adds one for deflate or expires (latter seems more unwise to me). See the supplemental configuration section of the generated config. This doesn't address mass-vhost companies failing to allow deflate because it's not in the no-args HTTPD ./configure , which sounds far-fetched to me. I can't recall a users@ or #httpd user implying being subjected to such a thing with their own build or with cheap hosting. -- Eric Covener cove...@gmail.com You seem to think large corporations are competent at web design/administration. -- Sent
Re: Fast by default
Let me preface ALL the remarks below with TWO statements... 1. I haven't done any research on these HTTP based Client/Server compression topics in quite some time. It is all, essentially, 'ancient history' for me but it still amazes me that some of the issues are, so many years later, still being 'guessed about' and no one has all the answers. 2. Let's not lose the TOPIC of this thread. The thread is about whether or not it's time to just turn mod_deflate ON by default in the 'safe' httpd.conf that ships with Apache. Regardless of disagreement on some of the internals and the remaining 'questions' I think it's clear to some now that this is NOT just an 'easy decision'. It's complicated. It WILL cause 'problems' for some installations and some environments. Bryan McQuade wrote... Kevin Kiley wrote... Don't forget the ongoing issue that if you ONLY vary on 'Accept-Encoding' then almost ALL browsers will then refuse to cache a response entity LOCALLY and the pain factor moves directly to the Proxy/Content Server(s). I don't think this is true for browsers in use today. Well, it's certainly true of the MSIE 6 I have 'in use today' on almost all of my Windows XP Virtual Machines that I use for testing. Also 'I don't think this is true' is certainly not 'I'm SURE this is not true'. Firefox will certainly cache responses with Vary: Accept-Encoding. I haven't done any testing with Firefox. Firefox wasn't even around when this first became an issue years ago. I'll take your word for it unless/until you can provide some Firefox specific test results page(s)that prove this to be true. The Mozilla/Firefox family has ALWAYS had a 'different' approach to how the client-end decompression gets done. That browser lineage chose to always use the local 'cache' as the place where the decompression takes place. That's why, when you use those browsers and you receive a compressed entity, you always get TWO cache files. One is simply the browser opening up a cache file to store the COMPRESSED version of the entity and the other is the DECOMPRESSED version. This is true even if the response is labeled 'Cache-control: private'. It will STILL 'cache' the response and ignore all 'Cache-Control:' directives but that's another 'bug' story altogether. They are simply doing all the decompression 'on disk' and using plain old file-based GUNZIP to get the job done so there HAS to be a 'cached copy' of the response regardless of any 'Cache-Control:' directives. It will also keep BOTH copies of the file around. The MSIE browser line will also use a cache-file ( sometimes ) for the decompression but, unlike the Mozilla lineage, MSIE will DELETE the initial compressed copy to avoid confusion. There used to be this weird-ass bug with Mozilla that would only show up if you tried to PRINT a decompressed page. The browser would forget that it had TWO disk files representing the compressed/uncompressed response and it would accidentally try to PRINT the COMPRESSED version. I certainly hope the Firefox branch of this source code line worked that little bug out. Eric Lawrence of the Internet Explorer team has a nice blog post that explains that in IE6 and later, Vary: Accept-Encoding are cached: http://blogs.msdn.com/b/ieinternals/archive/2009/06/17/vary-header-prevents-caching-in-ie.aspx. The 'EricLaw' MSDN Blog link you mention only says this about MSIE 6... [snip] Internet Explorer 6 Internet Explorer 6 will treat a response with a Vary header as completely uncacheable, unless the Vary header contains only the token User-Agent or Accept-Encoding. Hence, a subsequent request will be made unconditionally, resulting in a full re-delivery of the unchanged response. This results in a significant performance problem when Internet Explorer 6 encounters Vary headers. [/snip] This does NOT match my own research with MSIE 6 so the guy must be talking about some very specific BUILD version or 'hotpatched' version of MSIE 6. In case you missed it... here is the link to one of the discussions about this on the Apache mod_gzip forum which contains complete test results obtained using a kernel debugger with MSIE 4, 5 and 6... http://marc.info/?l=apache-modgzipm=103958533520502w=2 Eric also fails to mention that if you include MULTIPLE Vary: headers and/or multiple conditionals on the same 'Vary:' line ( as RFC 2616 says you are supposed to be able to do ) then MSIE 6 stops caching and treats those inbounds as the infamous 'Vary: *'. ( Vary: STAR ) I believe that last part is STILL TRUE even to this day which means it is STILL 'Non-RFC compliant'. This 'other issue' was also covered in my own MSIE 6 testing at the links above. Other variants of Vary do prevent caching in IE but Vary: Accept-Encoding is safe. According to EricLaw, yes... but see links and test results above. That is NOT what my own MSIE 6 testing showed. My testing on MSIE 6 only showed that ANY presence of ANY 'Vary:' header OTHER
Re: mod_deflate handling of empty initial brigade
On Tue, Jun 1, 2010 at 10:28 AM, Matthew Steele mdste...@google.com wrote: I went ahead and created a bug entry/patch to make the (trivial) change to mod_deflate to make it conform to the Guide to writing output filters: https://issues.apache.org/bugzilla/show_bug.cgi?id=49369 Brian, does this patch to mod_deflate fix your problem? Yes. (I'm still going to proceed with the same defensive logic in my own filter, though.) Thanks, -Brian
Re: Fast by default
All, I was once offered money to provide a high-performance Apache configuration file for a website. When I pointed out that I would need to come in, analyze their app and its performance, and then iteratively tune the config accordingly, I was given to understand that this was not necessary. Just send us the config, please. They were highly miffed when I didn't lay that particular flavor of golden egg for them. I actually got fired from that gig. On Jun 1, 2010, at 5:50 AM, Plüm, Rüdiger, VF-Group wrote: And others have argued well to leave it off by default. My personal opinion is that we should leave it disabled by default for the reasons (CPU, Proxies, Browser behaviour, ETAG problem) mentioned by others. I thought it isn't in the default build because it requires an external library that may not be on the system. If this is not of concern, we might as well turn it on in the default build. Same for mod_ssl. Generally, I think we see the build and runtime configuration as a starting point from which to create your own environment, not a canonical default that is right for all (or even most) users. Distributors (Red Hat et. al.) should (and do) build httpd according to the capabilities of their environment. If that environment includes libz and openssl, no reason why packagers shouldn't build those modules. Including those features is good for their customers. As others have pointed out, a lot of performance tuning is highly site and situation specific. Once again the default configuration file cannot be expected to cover all possible situations. Deflate, caching, load balancing, proxying, content segregation, etc. can only be optimally configured only in the context of the individual deployment. If there were a silver bullet to making the web server fast, don't you think we would have fired it some time in the past 15 years? There is no such thing. The only way to get a handle on it is to read the books, figure it out, or hire a consultant. But don't expect him to lay any golden performance eggs. S. -- Sander Temme scte...@apache.org PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
Re: Fast by default
On Tue, Jun 1, 2010 at 9:04 AM, William A. Rowe Jr. wr...@rowe-clan.net wrote: [...] Plus deflate may provide no benefit, and degrade performance, if the CPU utilization is a greater concern than bandwidth utilization. The CPU utilization is an interesting topic for me because I've been working on a related type of transform (minify of dynamic content) and I've gotten into the habit of measuring nanoseconds of processing time per byte of content. With that in mind, I just added in some instrumentation around the zlib calls in mod_deflate and found that the compression takes 30-60 ns/byte for buffers of a few KB. (This is with Apache 2.2.14 and zlib 1.2.3 on a 2.8GHz x64 processor.) To put that number in perspective, if you were to devote the equivalent of one CPU core worth of processing power on a web server host to compression, 30ns/byte means the host would be able to do real-time compression of ~266Mb/s of outbound traffic. Assume that your monthly bandwidth pricing is $10 per 95th percentile peak Mbps. Assume further that by turning on deflate you can reduce outbound bandwidth usage by 75% (i.e., you're getting 4:1 compression). Thus the CPU core that you've devoted completely to deflate processing will save you ~$2000 per month in bandwidth pricing. If the CPU core costs less than $24000 per year (amortized capital cost plus power, cooling, support, data center space, marginal cost of additional sysadmins needed to support each additional server, etc, etc), then you still come out ahead by turning on deflate. A few additional thoughts: 1. Speeding up the deflate implementation would be an unqualified win. Supposedly the recently-released zlib 1.2.5 is faster, but I don't have any data on it. 2. The best practice I've found for implementing compression in a web server is to do the compression in the server closest to the network edge. E.g., if you have a web server fronted by a proxy cache, do the compression dynamically in the proxy cache rather than the web server. That way the cache doesn't have to store multiple variants of each object. Similarly, if you're using a CDN for content that can benefit from gzip, ask your CDN if they can do conditional compression for non-problematic user-agents on-the-fly. -Brian
