Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
Quoting Bojan Smojver <[EMAIL PROTECTED]>: Just realised that the URL is wrong :-( So, here is the correct one: ftp://ftp.rexursive.com/pub/apache/counting_io_flush-2.0.43.patch.gz Bojan > On Fri, 2002-10-25 at 07:42, David Burry wrote: > > > I see.. ok, I'll keep waiting patiently... > > The patch for 2.0.43 is here: > > ftp://ftp.rexursive.com/pub/apache/counting_io_flush-2.0.43.patch > > You need to apply mod_logio patch for 2.0.43 first. > > Bojan > >
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
At 08:45 PM 10/24/2002 -0500, William A. Rowe, Jr. wrote:
>At 08:40 PM 10/24/2002, Bojan Smojver wrote:
>>Quoting David Burry <[EMAIL PROTECTED]>:
>>
>>> Excellent! I will perform some tests with that when I get a chance! You
>>> managed to get it working without breaking pipelining even? That's awesome!
>>
>>That's what I *think*, which has been known to deviate from the truth, from time
>>to time. However, I appreciate all input, especially results of the actual tests.
>
> I recall you had tested a ton of 'little files' pipelined.
>
> What might be more interesting is a 100MB download (over a fast pipe)
>which is entirely 'sendfile'd out. Apache would consider itself done with
>the request long before it was finished with the connection.
In case someone else wants to independently verify it...
The exact test I was doing was with a 70+ meg .tar.gz file both over a 100mbit
ethernet and a 1.5mbit DSL, starting and canceling it multiple times in Windoze
Internet Explorer 5 or 6 (which appears to effectively use byte range requests for
subsequent tries, by the way) and monitoring what was logged each time. This test
isn't super precise on the byte count (I did not bother to go comb my IE cache) but it
sure is obvious when it consistently logs the whole file size and I only received a
small fraction according to the IE progress bar... Also looking at the byte range
requests with %{Range}i makes it obvious how much IE received previously on each
subsequent try (IE appears to only request the part of the file it hasn't cached yet).
I was thinking of writing a script that did this in a more automated fashion...
perhaps contributing a test to the apache test suite when I figure that thing out...
;o)
Dave
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
At 09:38 PM 10/24/2002 -0400, Glenn wrote: >Have you looked at the %...X directive in Apache2? That's an interesting idea I hadn't thought of... it doesn't solve the chargeback issue but it's worth investigating for detecting successful downloads... Dave
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
> I recall you had tested a ton of 'little files' pipelined. > > What might be more interesting is a 100MB download (over a fast pipe) > which is entirely 'sendfile'd out. Apache would consider itself done with > the request long before it was finished with the connection. I tested with an 80 MB file and got different numbers depending on when I clicked stop in the browser. The testing was done on 127.0.0.1 (733 MHz Celeron box), so guess it qualifies as a fast pipe. Is that what you meant? Or did you mean pipelined large files? However, I do have a question for you (or anyone else that understands Apache 2.0 internal dymamics better then me). Currently, my patch fetches the optional function every time the connection is created (i.e. in core_pre_connection). I reckon this should actually be done in core_post_config, only once. Do you think that's safe (i.e. I'm guessing function pointers should be OK on graceful restarts, forks etc.).? Bojan
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
At 08:40 PM 10/24/2002, Bojan Smojver wrote: >Quoting David Burry <[EMAIL PROTECTED]>: > >> Excellent! I will perform some tests with that when I get a chance! You >> managed to get it working without breaking pipelining even? That's awesome! > >That's what I *think*, which has been known to deviate from the truth, from time >to time. However, I appreciate all input, especially results of the actual tests. Bojon, I recall you had tested a ton of 'little files' pipelined. What might be more interesting is a 100MB download (over a fast pipe) which is entirely 'sendfile'd out. Apache would consider itself done with the request long before it was finished with the connection. Just a thought. Bill
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
On Thu, Oct 24, 2002 at 05:25:46PM -0700, David Burry wrote:
> Excellent! I will perform some tests with that when I get a chance! You managed to
>get it working without breaking pipelining even? That's awesome!
>
> Not meaning to belittle Bojan's hard work, but for my purposes mod_logio values are
>not as good as %b would be if %b worked properly... what I ideally need is the byte
>sent count without the headers... using Bojan's module I can get approximate results
>but they will be a hair off because they include headers... My main purpose is to
>detect if and when several meg files have been downloaded all the way vs. if they
>were cut off in the middle, including if a given user uses some byte-ranging download
>manager that lets you pause and restart... We also use it for chargebacks to the
>various departments for bandwidth usage (in this case mod_logio would of course be
>more accurate than %b though)... We actually had to fudge some of our statistics
>(duplicated nearby days' data with similar overall throughputs) due to us not
>catching the problem with Apache 2.0 soon enough...
Have you looked at the %...X directive in Apache2?
http://httpd.apache.org/docs-2.0/mod/mod_log_config.html
%...X: Connection status when response is completed.
'X' = connection aborted before the response completed.
'+' = connection may be kept alive after the response is sent.
'-' = connection will be closed after the response is sent.
(This directive was %...c in late versions of Apache 1.3, but
this conflicted with the historical ssl %...{var}c syntax.)
I don't know if this works in Apache2, but this is what the docs say.
Discussion of Apache2 design on this list would suggest that it might
suffer from the same limitation that %b currently exhibits.
(In the past, I used the %c in Apache 1.3 with success when tallying
whether or not a download of a .cab installer completed successfully.
And I didn't have to know the size of the file for comparison. Nice!)
-Glenn
> At 09:15 AM 10/25/2002 +1000, Bojan Smojver wrote:
> >On Fri, 2002-10-25 at 07:42, David Burry wrote:
> >
> >> I see.. ok, I'll keep waiting patiently...
> >
> >The patch for 2.0.43 is here:
> >
> >ftp://ftp.rexursive.com/pub/apache/counting_io_flush-2.0.43.patch
> >
> >You need to apply mod_logio patch for 2.0.43 first.
> >
> >Bojan
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
Quoting David Burry <[EMAIL PROTECTED]>: > Excellent! I will perform some tests with that when I get a chance! You > managed to get it working without breaking pipelining even? That's awesome! That's what I *think*, which has been known to deviate from the truth, from time to time. However, I appreciate all input, especially results of the actual tests. Bojan
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
Excellent! I will perform some tests with that when I get a chance! You managed to get it working without breaking pipelining even? That's awesome! Not meaning to belittle Bojan's hard work, but for my purposes mod_logio values are not as good as %b would be if %b worked properly... what I ideally need is the byte sent count without the headers... using Bojan's module I can get approximate results but they will be a hair off because they include headers... My main purpose is to detect if and when several meg files have been downloaded all the way vs. if they were cut off in the middle, including if a given user uses some byte-ranging download manager that lets you pause and restart... We also use it for chargebacks to the various departments for bandwidth usage (in this case mod_logio would of course be more accurate than %b though)... We actually had to fudge some of our statistics (duplicated nearby days' data with similar overall throughputs) due to us not catching the problem with Apache 2.0 soon enough... Dave At 09:15 AM 10/25/2002 +1000, Bojan Smojver wrote: >On Fri, 2002-10-25 at 07:42, David Burry wrote: > >> I see.. ok, I'll keep waiting patiently... > >The patch for 2.0.43 is here: > >ftp://ftp.rexursive.com/pub/apache/counting_io_flush-2.0.43.patch > >You need to apply mod_logio patch for 2.0.43 first. > >Bojan
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
On Fri, 2002-10-25 at 07:42, David Burry wrote: > I see.. ok, I'll keep waiting patiently... The patch for 2.0.43 is here: ftp://ftp.rexursive.com/pub/apache/counting_io_flush-2.0.43.patch You need to apply mod_logio patch for 2.0.43 first. Bojan
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
- Original Message - From: "Bojan Smojver" <[EMAIL PROTECTED]> > On Fri, 2002-10-25 at 03:31, David Burry wrote: > > Is it possible to get some of the fixes to mod_logio committed? Wouldn't > > everyone agree that the current logging of the outgoing bytes is incorrect > > behavior? Currently it logs the full file size (plus headers) even if it > > gets cut off in the middle, instead of the actual number of bytes sent. > > I've seen several patches to fix this but very little comment on it... I've > > seen lots of comments that it can't be done without major rearchitecting, > > but Bojan seems to have done it without that by breaking pipelining, am I > > correct? > > Actually, the last patch I sent contains one snag I'm still working on. > It breaks the core's connection configuration structure, which gets > attached to c->conn_config. However, I think I can get around that by > using an optional function. As the matter of fact, I'm working on it > right now. I see.. ok, I'll keep waiting patiently... > > Since I depend on correct outgoing byte count logging to see how many people > > sucessfully download files, I can live with broken pipelining for now in > > 2.0, currently I've had to roll back to Apache 1.3 and put in 3 times as > > many machines (12 instead of 4) I'd really like to return those 8 > > borrowed machines someday and be able to upgrade to 2.0... but can't do that > > in the current broken log state. > > Glad to hear Apache 2.0 makes a huge performance difference. Not so glad > to hear you had to resort to going back to 1.3. The only thing I can > promise is a patch using optional function (this should guarantee > compatibility of core between 43 and 44 and no MMN bump) during the day > (Sydney time). It's up to the committers to review and, if they like, > commit. The memory savings is quite significant, and I'll admit that the 8 extra machines are smaller than the original 4 so it's not exactly 3 times better cpu-wise... the memory caching is where the largest savings comes with disk IO, we have a very high traffic site--usually 3 terabytes transferred per day, last few days have been more like 5 terabytes due to a new release. > PS. By the number of messages on the list I'm guessing committers must > be rather busy on their real jobs these days. Unfortunately there is no > way of speeding things up, given this is volunteer effort. Unless, of > course, you decide to bribe some of them ;-) Not exactly the same thing as bribing a commit, but this could get similar results: My manager's manager is actually not opposed to hiring a contractor to fix this... anyone want a temporary job? I don't know if this is the right place to say such things, tell me if there's a better place. When you've got millions of dollars worth of sales depending on an open source project, throwing a little at the open source project isn't such a big deal... I'd gladly do it myself with my company's blessing (on the clock, not volunteer) but I'm not a very experienced C programmer yet, more of a Perl hacker and applications architect so far. This little paragraph had better not get me too flooded with resumes or flames or I'm going to feel dumb, whatever you do don't spam this list with personal replies! ;o) Dave
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
Is it possible to get some of the fixes to mod_logio committed? Wouldn't everyone agree that the current logging of the outgoing bytes is incorrect behavior? Currently it logs the full file size (plus headers) even if it gets cut off in the middle, instead of the actual number of bytes sent. I've seen several patches to fix this but very little comment on it... I've seen lots of comments that it can't be done without major rearchitecting, but Bojan seems to have done it without that by breaking pipelining, am I correct? I also wish that %b would be fixed in a similar manner but I haven't seen any patches for that (or comments about it). Wouldn't everyone agree that it too should log actual bytes sent not just the full file size every time? Apache 2.0 should do everything that 1.3 did, so this logging issue really should be considered a bug, right? Since I depend on correct outgoing byte count logging to see how many people sucessfully download files, I can live with broken pipelining for now in 2.0, currently I've had to roll back to Apache 1.3 and put in 3 times as many machines (12 instead of 4) I'd really like to return those 8 borrowed machines someday and be able to upgrade to 2.0... but can't do that in the current broken log state. Dave
Re: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
On Fri, 2002-10-25 at 03:31, David Burry wrote: > Is it possible to get some of the fixes to mod_logio committed? Wouldn't > everyone agree that the current logging of the outgoing bytes is incorrect > behavior? Currently it logs the full file size (plus headers) even if it > gets cut off in the middle, instead of the actual number of bytes sent. > I've seen several patches to fix this but very little comment on it... I've > seen lots of comments that it can't be done without major rearchitecting, > but Bojan seems to have done it without that by breaking pipelining, am I > correct? Actually, the last patch I sent contains one snag I'm still working on. It breaks the core's connection configuration structure, which gets attached to c->conn_config. However, I think I can get around that by using an optional function. As the matter of fact, I'm working on it right now. > I also wish that %b would be fixed in a similar manner but I haven't seen > any patches for that (or comments about it). Wouldn't everyone agree that > it too should log actual bytes sent not just the full file size every time? > Apache 2.0 should do everything that 1.3 did, so this logging issue really > should be considered a bug, right? I agree with you on this one as well. But at this point I'm unsure how to fix this one. > Since I depend on correct outgoing byte count logging to see how many people > sucessfully download files, I can live with broken pipelining for now in > 2.0, currently I've had to roll back to Apache 1.3 and put in 3 times as > many machines (12 instead of 4) I'd really like to return those 8 > borrowed machines someday and be able to upgrade to 2.0... but can't do that > in the current broken log state. Glad to hear Apache 2.0 makes a huge performance difference. Not so glad to hear you had to resort to going back to 1.3. The only thing I can promise is a patch using optional function (this should guarantee compatibility of core between 43 and 44 and no MMN bump) during the day (Sydney time). It's up to the committers to review and, if they like, commit. Bojan PS. By the number of messages on the list I'm guessing committers must be rather busy on their real jobs these days. Unfortunately there is no way of speeding things up, given this is volunteer effort. Unless, of course, you decide to bribe some of them ;-)
RE: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
> -Original Message- > From: Rodent of Unusual Size [mailto:Ken.Coar@;Golux.Com] > Sent: Thursday, October 24, 2002 06:45 > To: Apache HTTP server developers > Subject: [STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002 > > > APACHE 2.0 STATUS: > -*-text-*- > Last modified at [$Date: 2002/10/22 17:01:44 $] Hi there! I would be happy to see Bugzilla #7617 be applied to the Apache 2.0, as I just have submitted a patch for 2.0.40 with regard to that annoying bug. Or if my patch is insufficient, at least someone with the proper knowledge should open a PR for Apache 2.0 about that behaviour (I do not feel like the right person to do so). -- [EMAIL PROTECTED] GSM +358-40-767 8282 Oy Arrak Software Ab http://www.arrak.fi
[STATUS] (httpd-2.0) Wed Oct 23 23:45:19 EDT 2002
APACHE 2.0 STATUS: -*-text-*- Last modified at [$Date: 2002/10/22 17:01:44 $] Release: 2.0.44 : in development 2.0.43 : rolled October 2, 2002 2.0.42 : released September 24, 2002 as GA. 2.0.41 : rolled September 16, 2002. not released. 2.0.40 : released August 9, 2002 as GA. 2.0.39 : released June 17, 2002 as GA. 2.0.38 : rolled June 16, 2002. not released. 2.0.37 : rolled June 11, 2002. not released. 2.0.36 : released May 6, 2002 as GA. 2.0.35 : released April 5, 2002 as GA. 2.0.34 : tagged March 26, 2002. 2.0.33 : tagged March 6, 2002. not released. 2.0.32 : released Feburary 16, 2002 as beta. 2.0.31 : rolled Feburary 1, 2002. not released. 2.0.30 : tagged January 8, 2002. not rolled. 2.0.29 : tagged November 27, 2001. not rolled. 2.0.28 : released November 13, 2001 as beta. 2.0.27 : rolled November 6, 2001 2.0.26 : tagged October 16, 2001. not rolled. 2.0.25 : rolled August 29, 2001 2.0.24 : rolled August 18, 2001 2.0.23 : rolled August 9, 2001 2.0.22 : rolled July 29, 2001 2.0.21 : rolled July 20, 2001 2.0.20 : rolled July 8, 2001 2.0.19 : rolled June 27, 2001 2.0.18 : rolled May 18, 2001 2.0.17 : rolled April 17, 2001 2.0.16 : rolled April 4, 2001 2.0.15 : rolled March 21, 2001 2.0.14 : rolled March 7, 2001 2.0a9 : released December 12, 2000 2.0a8 : released November 20, 2000 2.0a7 : released October 8, 2000 2.0a6 : released August 18, 2000 2.0a5 : released August 4, 2000 2.0a4 : released June 7, 2000 2.0a3 : released April 28, 2000 2.0a2 : released March 31, 2000 2.0a1 : released March 10, 2000 Please consult the following STATUS files for information on related projects: * srclib/apr/STATUS * srclib/apr-util/STATUS * docs/STATUS Contributors looking for a mission: * just do an egrep on "TODO" and see what's there CURRENT RELEASE NOTES: RELEASE SHOWSTOPPERS: CURRENT VOTES: * httpd-std.conf and friends a) httpd-std.conf should be tailored by install (from src or binbuild) even if user has existing httpd.conf +1: trawick, slive, gregames, ianh, Ken b) tailored httpd-std.conf should be copied by install to sysconfdir/examples -0: striker c) tailored httpd-std.conf should be installed to sysconfdir/examples or manualdir/exampleconf/ +1: slive, trawick, Ken d) Installing a set of default config files when upgrading a server doesn't make ANY sense at all. +1: striker ianh - medium/big sites don't use 'standard config' anyway, as it usually needs major customizations -1: Ken * If the parent process dies, should the remaining child processes "gracefully" self-terminate. Or maybe we should make it a runtime option, or have a concept of 2 parent processes (one being a "hot spare"). See: Message-ID: <[EMAIL PROTECTED]> Self-destruct: Ken, Martin Not self-destruct: BrianP, Ian, Cliff, BillS Make it runtime configurable: Aaron, Jim, Justin Have 2 parents: +1: Jim -1: Justin, wrowe [for 2.0] +0: Martin (while standing by, could it do something useful?) * Make the worker MPM the default MPM for threaded Unix boxes. +1: Justin, Ian, Cliff, BillS, striker +0: BrianP, Aaron (mutex contention is looking better with the latest code, let's continue tuning and testing) -0: Lars RELEASE NON-SHOWSTOPPERS BUT WOULD BE REAL NICE TO WRAP THESE UP: * There is a bug in how we sort some hooks, at least the pre-config hook. The first time we call the hooks, they are in the correct order, but the second time, we don't sort them correctly. Currently, the modules/http/config.m4 file has been renamed to modules/http/config2.m4 to work around this problem, it should moved back when this is fixed. OtherBill offers that this is a SERIOUS problem. We do not sort correctly by the ordering arguments passed to the register hook functions. This was proven when I reordered the open_logs hook to attempt to open the error logs prior to the access logs. Possibly the entire sorting code needs to be refactored. * pipes deadlock on all platforms with limited pipe buffers (e.g. both Linux and Win32, as opposed to only Win32 on 1.3). The right solution is either GStein's proposal for a "CGI Brigade", or OtherBill's proposal for "Poll Buckets" for "Polling Filter Chains". * server pushed CGI's not working. This might be an interaction with the above pipes deadlock issue. PR: 8482 Message-ID: <[EMAIL PROTECTED]> *
