Re: Apache logging
On Fri, 10 Sep 2010, Sergio Junqueira wrote: I have a suggestion for the developers of Apache related to mod_log_config or mod_log_forensics: 1) To allow mod_log_config to write the log file with a first log entry with basic information about the request before it's processed further (that is, after receiving the headers). The second log entry (the current logging format) would be written after the request processing. A "log ID", just like the "forensic ID", should be available on both entries. 2)- Or to allow mod_log_forensic to be configured not to write all headers to disk. For instance, we should be able to configure it just like we do with LogFormat on mod_log_config. Or at least allow us to choose to write just the 1st request line, without all the headers. I think 2) would be a reasonable feature request for mod_log_forensic and would be easy to implement. However, I am interested about your actual use case. Do you need the first entry to determine which request may have caused httpd to crash or is there a different reason? If you are generally interested in better logging in httpd, you may want to read and possibly comment about some new features in 2.3/2.4: http://httpd.apache.org/docs/trunk/mod/core.html#errorlogformat http://httpd.apache.org/docs/trunk/mod/core.html#loglevel Cheers, Stefan
Bug report for Apache httpd-1.3 [2010/09/12]
+---+ | Bugzilla Bug ID | | +-+ | | Status: UNC=Unconfirmed NEW=New ASS=Assigned| | | OPN=ReopenedVER=Verified(Skipped Closed/Resolved) | | | +-+ | | | Severity: BLK=Blocker CRI=Critical REG=Regression MAJ=Major | | | | MIN=Minor NOR=NormalENH=Enhancement TRV=Trivial | | | | +-+ | | | | Date Posted | | | | | +--+ | | | | | Description | | | | | | | |10744|New|Nor|2002-07-12|suexec might fail to open log file| |10747|New|Maj|2002-07-12|ftp SIZE command and 'smart' ftp servers results i| |10760|New|Maj|2002-07-12|empty ftp directory listings from cached ftp direc| |14518|Opn|Reg|2002-11-13|QUERY_STRING parts not incorporated by mod_rewrite| |16013|Opn|Nor|2003-01-13|Fooling mod_autoindex + IndexIgnore | |16631|Inf|Min|2003-01-31|.htaccess errors logged outside the virtual host l| |17318|Inf|Cri|2003-02-23|Abend on deleting a temporary cache file if proxy | |19279|Inf|Min|2003-04-24|Invalid chmod options in solaris build| |21637|Inf|Nor|2003-07-16|Timeout causes a status code of 200 to be logged | |21777|Inf|Min|2003-07-21|mod_mime_magic doesn't handle little gif files| |21975|Opn|Nor|2003-07-29|mod_rewrite RewriteMap from external program gets | |22618|New|Maj|2003-08-21|MultiViews invalidates PATH_TRANSLATED if cgi-wrap| |25057|Inf|Maj|2003-11-27|Empty PUT access control in .htaccess overrides co| |26126|New|Nor|2004-01-14|mod_include hangs with request body | |26152|Ass|Nor|2004-01-15|Apache 1.3.29 and below directory traversal vulner| |26790|New|Maj|2004-02-09|error deleting old cache file | |29257|Opn|Nor|2004-05-27|Problem with apache-1.3.31 and mod_frontpage (dso,| |29498|New|Maj|2004-06-10|non-anonymous ftp broken in mod_proxy | |29538|Ass|Enh|2004-06-12|No facility used in ErrorLog to syslog| |30207|New|Nor|2004-07-20|Piped logs don't close read end of pipe | |30877|New|Nor|2004-08-26|htpasswd clears passwd file on Sun when /var/tmp i| |30909|New|Cri|2004-08-28|sporadic segfault resulting in broken connections | |31975|New|Nor|2004-10-29|httpd-1.3.33: buffer overflow in htpasswd if calle| |32078|New|Enh|2004-11-05|clean up some compiler warnings | |32539|New|Trv|2004-12-06|[PATCH] configure --enable-shared= brocken on SuSE| |32974|Inf|Maj|2005-01-06|Client IP not set | |33086|New|Nor|2005-01-13|unconsistency betwen 404 displayed path and server| |33495|Inf|Cri|2005-02-10|Apache crashes with "WSADuplicateSocket failed for| |33772|New|Nor|2005-02-28|inconsistency in manual and error reporting by sue| |33875|New|Enh|2005-03-07|Apache processes consuming CPU| |34108|New|Nor|2005-03-21|mod_negotiation changes mtime to mtime of Document| |34114|New|Nor|2005-03-21|Apache could interleave log entries when writing t| |34404|Inf|Blk|2005-04-11|RewriteMap prg can not handle fpout | |34571|Inf|Maj|2005-04-22|Apache 1.3.33 stops logging vhost| |34573|Inf|Maj|2005-04-22|.htaccess not working / mod_auth_mysql| |35424|New|Nor|2005-06-20|httpd disconnect in Timeout on CGI| |35439|New|Nor|2005-06-21|Problem with remove "/../" in util.c and mod_rewri| |35547|Inf|Maj|2005-06-29|Problems with libapreq 1.2 and Apache::Cookie | |3|New|Nor|2005-06-30|Can't find DBM on Debian Sarge| |36375|Opn|Nor|2005-08-26|Cannot include http_config.h from C++ file| |37166|New|Nor|2005-10-19|Under certain conditions, mod_cgi delivers an empt| |37252|New|Reg|2005-10-26|gen_test_char reject NLS string | |38989|New|Nor|2006-03-15|restart + piped logs stalls httpd for 24 minutes (| |39104|New|Enh|2006-03-25|[FR] fix build with -Wl,--as-needed | |39287|New|Nor|2006-04-12|Incorrect If-Modified-Since validation (due to syn| |39937|New|Nor|2006-06-30|Garbage output if README.html is gzipped or compre| |40224|Ver|Nor|2006-08-10|System time crashes Apache @year 2038 (win32 only?| |41279|New|Nor|2007-01-02|Apache 1.3.37 htpasswd is vulnerable to buffer ove| |42355|New|Maj|2007-05-08|Apache 1.3 permits non-rfc HTTP error code >= 600 | |43626|New|Maj|2007-10-15|r->path_info returning invalid value | |44768|New|Blk|2008-04-07|Server suddenly reverted to showing test page only| |44926|
Re: svn commit: r992625 - in /httpd/httpd/trunk: CHANGES modules/cache/cache_storage.c modules/cache/cache_util.c modules/cache/mod_cache.h
On 05 Sep 2010, at 7:05 PM, Ruediger Pluem wrote: Nitpicking: IMHO we are changing a public API. Where is the minor bump? Thanks for the catch, it's in r996311. Regards, Graham --
Re: mod_cache: store_body() bites off more than it can chew
On 06 Sep 2010, at 11:00 PM, Paul Querna wrote: Isn't this problem an artifact of how all bucket brigades work, and is present in all output filter chains? An output filter might be called multiple times, but a single bucket can still contain a 4gb chunk easily. It seems to me it would be better to think about this holistically down the entire output filter chain, rather than building in special case support for this inside mod_cache's internal methods? In the cache case, thinking about it a bit the in and out brigades are probably unavoidable, as the cache is a special case in that it wants to write the data twice, once to the cache, a second time to the rest of the filter stack. Right now, the cache is forced to read the complete brigade to cache it, no option to give up early. And the cache has no choice but to keep the brigade buckets in the brigade so that they can be passed a second time up the filter stack, no deleting buckets as you go like you normally would. Read one 4GB file bucket in the cache, and in the process the file bucket gets morphed into 1/2 million heap buckets, oops. With two brigades, one in, one out, the in brigade can have the buckets removed as they are consumed, as normal, and moved to the out brigade. The cache can quit at any time, and the code following knows what data to write to the network (out), and what data to loop round and resend to the cache (in). The cache provider could choose to quit and ask to be called again either because writing took too long, or too much data was read (and in the process became heap buckets), either reason is fine. That said, following on your suggestion of thinking about this in the general sense, it would be really nice if the filter stack had the option to say "I have bitten off as much of the brigade as I am prepared to chew on right now, and the leftovers are still in the brigade, can you call me back with this data, maybe with more data added, and I'll try swallow some more?". In theory, that would mean all handlers (or entities that sent data) would no longer be allowed to make the blind assumption that the filter stack was willing to consume every possible set of buckets the handler wanted to send, and that the stack had the right to go "I'm full, give me a second to chew on this". This wouldn't need separate brigades, probably just a return code that meant EAGAIN, and that was expected to be honoured by handlers. Regards, Graham --
mod_disk_cache: CacheMaxFileSize and CacheMinFileSize per directory
Hi all, The CacheMinFileSize and CacheMaxFileSize directives in mod_disk_cache are currently set per server, which seems to be historical from the time before mod_cache could be added as a normal handler / specifically placed filter. This stops an administrator applying a cache size policy per directory or location. The fix is to move the directives to the per-directory config. This will fix the problem, while being backwards compatible with existing configurations. A further issue is that of namespace use, in theory these directives (and other directives in the mod_disk_cache module) should be prefixed with "CacheDisk" or "DiskCache" instead of "Cache". Thoughts? Regards, Graham --
Re: Apache logging
>Do you need the first entry to determine which request may have caused httpd >to >crash or is there a different reason? Mod_log_forensics writes the log record as soon as it is received. Mod_log_config writes the log record after the response is available. I don´t want to miss information about any request. Its important to identify all incoming requests, no matter if they fail or succeed. I would like to trace them having a small log record from the incoming requests. Useful to determine which request may have caused httpd to crash, as you mentioned, and to have 100% guarantee that all incoming requests were logged before Apache started processing it. Does mod_log_config always writes a log record for all incoming requests, no matter where or when they failed in the processing chain? If this is the case, mod_log_config provides the %D format to determine when the incoming request was received and it seems the current mod_log_config record would provide the information I need. Thanks, Sergio. From: Stefan Fritsch <@sfritsch.de> To: Sergio Junqueira <@yahoo.com> Cc: dev@httpd.apache.org Sent: Sun, September 12, 2010 7:18:07 AM Subject: Re: Apache logging On Fri, 10 Sep 2010, Sergio Junqueira wrote: > I have a suggestion for the developers of Apache related to mod_log_config or > mod_log_forensics: > > 1) To allow mod_log_config to write the log file with a first log entry with > basic information about the request before it's processed further (that is, > after receiving the headers). The second log entry (the current logging format) > would be written after the request processing. A "log ID", just like the > "forensic ID", should be available on both entries. > > 2)- Or to allow mod_log_forensic to be configured not to write all headers to > disk. For instance, we should be able to configure it just like we do with > LogFormat on mod_log_config. Or at least allow us to choose to write just the > 1st request line, without all the headers. I think 2) would be a reasonable feature request for mod_log_forensic and would be easy to implement. However, I am interested about your actual use case. Do you need the first entry to determine which request may have caused httpd to crash or is there a different reason? If you are generally interested in better logging in httpd, you may want to read and possibly comment about some new features in 2.3/2.4: http://httpd.apache.org/docs/trunk/mod/core.html#errorlogformat http://httpd.apache.org/docs/trunk/mod/core.html#loglevel Cheers, Stefan
RE: Apache logging
I'm not sure it is related, but I'd like to know the most efficient way to debug error_log entries such as the following. In the first case, I presume that the referrer was absent or was unable to be read. Ideally a "conditional" forensics log (i.e. only on error response codes) would probably do it, but I don't think that's possible, right? Thanks, David [Mon Sep 13 04:12:25 2010] [error] [client 71.225.73.189] request failed: error reading the headers [Mon Sep 13 04:25:22 2010] [error] [client 210.204.226.18] request failed: error reading the headers, referer: http://foo.com?dtm_com=28&dtm_cmagic=d3b1fb&dtm_fid=101&dtm_format=5&cli_pro mo_id=1&dtmc_ver=3&dtm_cid=2366&dtmc_u2=/category&dtmc_u3=26047&dtmc_u4=null &dtmc_u5=null&dtmc_u9=gp%253Abrowse%253Ababy%253AToddler%2520Girl%2520%25281 -5%2520yrs%2529%253ASale%253ASweaters&dtmc_u10=http%253A//blah.com/browse/ca tegory.do%253Fcid%253D26047&dtmc_transaction_id=1%3F&dtmc_type=090&dtmc_cat= 037&dtmc_ref=http%3A//blah.com/browse/category.do%3Fcid%3D26047&dtmc_url=htt p%3A//fls.doubleclick.net/activityi%3Bsrc%3D2840522%3Btype%3D090%3Bcat%3D037 %3Bqty%3D1%3Btran%3Dnull%3Bu%3DUSD%3Bu2%3D/category%3Bu3%3D26047%3Bu4%3Dnull %3Bu5%3Dnull%3Bu9%3Dgp%253Abrowse%253A%253AToddler%2520Girl%2520%25281-5%252 0yrs%2529%253ASale%253ASweaters%3Bu10%3Dhttp%253A//blah.com/browse/category. do%253Fcid%253D26047%3Bord%3D1%3F& From: Sergio Junqueira Sent: Sunday, September 12, 2010 8:35 PM To: dev@httpd.apache.org Subject: Re: Apache logging >Do you need the first entry to determine which request may have caused httpd to crash or is there a different reason? Mod_log_forensics writes the log record as soon as it is received. Mod_log_config writes the log record after the response is available. I don´t want to miss information about any request. Its important to identify all incoming requests, no matter if they fail or succeed. I would like to trace them having a small log record from the incoming requests. Useful to determine which request may have caused httpd to crash, as you mentioned, and to have 100% guarantee that all incoming requests were logged before Apache started processing it. Does mod_log_config always writes a log record for all incoming requests, no matter where or when they failed in the processing chain? If this is the case, mod_log_config provides the %D format to determine when the incoming request was received and it seems the current mod_log_config record would provide the information I need. Thanks, Sergio. Cc: dev@httpd.apache.org Sent: Sun, September 12, 2010 7:18:07 AM Subject: Re: Apache logging On Fri, 10 Sep 2010, Sergio Junqueira wrote: > I have a suggestion for the developers of Apache related to mod_log_config or > mod_log_forensics: > > 1) To allow mod_log_config to write the log file with a first log entry with > basic information about the request before it's processed further (that is, > after receiving the headers). The second log entry (the current logging format) > would be written after the request processing. A "log ID", just like the > "forensic ID", should be available on both entries. > > 2)- Or to allow mod_log_forensic to be configured not to write all headers to > disk. For instance, we should be able to configure it just like we do with > LogFormat on mod_log_config. Or at least allow us to choose to write just the > 1st request line, without all the headers. I think 2) would be a reasonable feature request for mod_log_forensic and would be easy to implement. However, I am interested about your actual use case. Do you need the first entry to determine which request may have caused httpd to crash or is there a different reason? If you are generally interested in better logging in httpd, you may want to read and possibly comment about some new features in 2.3/2.4: http://httpd.apache.org/docs/trunk/mod/core.html#errorlogformat http://httpd.apache.org/docs/trunk/mod/core.html#loglevel Cheers, Stefan
Re: svn commit: r996395 - in /httpd/httpd/trunk: CHANGES docs/manual/mod/mod_disk_cache.xml include/ap_mmn.h modules/cache/mod_cache.c modules/cache/mod_cache.h modules/cache/mod_disk_cache.c modules/
On 09/13/2010 01:16 AM, minf...@apache.org wrote:
> Author: minfrin
> Date: Sun Sep 12 23:16:49 2010
> New Revision: 996395
>
> URL: http://svn.apache.org/viewvc?rev=996395&view=rev
> Log:
> mod_cache: Change the signature of the store_body() provider function
> within the mod_cache provider interface to support an "in" brigade
> and an "out" brigade instead of just a single input brigade. This
> gives a cache provider the option to consume only part of the brigade
> passed to it, rather than the whole brigade as was required before.
> This fixes an out of memory and a request timeout condition that would
> occur when the original document was a large file. Update the
> mod_disk_cache provider implementation to take into account the new API.
> Introduce CacheReadSize and CacheReadTime directives to mod_disk_cache
> to control the amount of data to attempt to cache before sending the
> data on to the client in the "out" brigade.
>
> Modified:
> httpd/httpd/trunk/CHANGES
> httpd/httpd/trunk/docs/manual/mod/mod_disk_cache.xml
> httpd/httpd/trunk/include/ap_mmn.h
> httpd/httpd/trunk/modules/cache/mod_cache.c
> httpd/httpd/trunk/modules/cache/mod_cache.h
> httpd/httpd/trunk/modules/cache/mod_disk_cache.c
> httpd/httpd/trunk/modules/cache/mod_disk_cache.h
>
> Modified: httpd/httpd/trunk/modules/cache/mod_disk_cache.c
> URL:
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/cache/mod_disk_cache.c?rev=996395&r1=996394&r2=996395&view=diff
> ==
> --- httpd/httpd/trunk/modules/cache/mod_disk_cache.c (original)
> +++ httpd/httpd/trunk/modules/cache/mod_disk_cache.c Sun Sep 12 23:16:49 2010
> @@ -1028,22 +1031,65 @@ static apr_status_t store_body(cache_han
> }
> dobj->file_size = 0;
> }
> +if (!dobj->bb) {
> +dobj->bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
> +}
> +if (!dobj->offset) {
> +dobj->offset = dconf->readsize;
> +}
> +if (!dobj->timeout && dconf->readtime) {
> +dobj->timeout = apr_time_now() + dconf->readtime;
> +}
>
> -for (e = APR_BRIGADE_FIRST(bb);
> - e != APR_BRIGADE_SENTINEL(bb);
> - e = APR_BUCKET_NEXT(e))
> -{
> +if (dobj->offset) {
> +apr_brigade_partition(in, dobj->offset, &e);
> +}
> +
> +while (APR_SUCCESS == rv && !APR_BRIGADE_EMPTY(in)) {
> const char *str;
> apr_size_t length, written;
> +
> +e = APR_BRIGADE_FIRST(in);
> +
> +/* have we seen eos yet? */
> +if (APR_BUCKET_IS_EOS(e)) {
> +seen_eos = 1;
> +APR_BUCKET_REMOVE(e);
> +APR_BRIGADE_CONCAT(out, dobj->bb);
> +APR_BRIGADE_INSERT_TAIL(out, e);
> +continue;
Can't this lead to a situation where buckets that follow an EOS bucket (the
only ones
I can think of are Metabuckets) get swallowed forever by mod_disk_cache?
These possible Metabuckets will be swallowed and added to dobj->bb, but never
put
in the out brigade. So shouldn't we just CONCAT the remaining part of in to out
and
leave?
Regards
Rüdiger
Re: Apache logging
- "Sergio Junqueira" wrote: > >Do you need the first entry to determine which request may have > caused httpd to crash or is there a different reason? > > > Mod_log_forensics writes the log record as soon as it is received. > Mod_log_config writes the log record after the response is available. > I don´t want to miss information about any request. Its important to > identify all incoming requests, no matter if they fail or succeed. > > > I would like to trace them having a small log record from the incoming > requests. Useful to determine which request may have caused httpd to > crash, as you mentioned, and to have 100% guarantee that all incoming > requests were logged before Apache started processing it. Have you already exhausted other methods such as mod_whatkilledus? bye, i -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: i.ga...@brainsware.org URL: http://brainsware.org/
Re: mod_disk_cache: CacheMaxFileSize and CacheMinFileSize per directory
On 09/13/2010 01:29 AM, Graham Leggett wrote: > Hi all, > > The CacheMinFileSize and CacheMaxFileSize directives in mod_disk_cache > are currently set per server, which seems to be historical from the time > before mod_cache could be added as a normal handler / specifically > placed filter. This stops an administrator applying a cache size policy > per directory or location. > > The fix is to move the directives to the per-directory config. This will > fix the problem, while being backwards compatible with existing > configurations. > > A further issue is that of namespace use, in theory these directives > (and other directives in the mod_disk_cache module) should be prefixed > with "CacheDisk" or "DiskCache" instead of "Cache". Thoughts? +1 to the name and I guess we could break the naming between 2.2 and 2.4, possibly with a more precise error message that says the new name of the directive instead of just the typical unknown message. Regards Rüdiger
