RE: 2.0.34 - erratic behavior with autoindexes
Okay, I have had more time to investigate the problem. This is kind of strange. First of all, I have included a patch with this post that is absolutely 100% wrong. The only purpose of the patch is to show the problem, and hopefully prove that I have isolated the problem correctly. I wasn't able to reproduce the exact problem, but I traced the problem, and I think I saw the cause, even if I didn't see the symptoms. The problem is that we have a main request that does NOT have the OLD_WRITE filter. Then we create the sub-request, which means that the SUB_REQ_EOS filter is added without the OLD_WRITE filter. Once we have created the sub-request (notice we haven't run it yet), we write to the next filter using ap_r* functions. This will insert the OLD_WRITE filter, and the data will be buffered in the filter. Finally, we actually run the sub-request. The important thing here, is that we have never actually put the OLD_WRITE filter into the sub-request, because it was added after we created the sub-request. This means that all of the data from the sub-request is written BEFORE the pre-amble. Because of that, we get VERY invalid HTML. The patch below forces mod_autoindex to insert the OLD_WRITE filter before the sub-request is created. As I have said, this is the WRONG solution. I have tested it at home, and it does force things to be written in the correct order. My hope is that this patch can be tested on daedalus to ensure that it will work to solve the problem we have seen. _IF_ this does work, then I can think of three possible solutions to fix the problem for real. 1) Make sure we write the pre-amble before we create the sub-request. The logic is possible, but it is kind of ugly. I definitely won't have time to write that patch until Thursday night, so I hope somebody else can beat me to it. 2) The second option is to modify ap_run_sub_req(), so that it detects this case and inserts the OLD_WRITE filter for the sub-request. This will be exceedingly ugly, and I would prefer to not use this method. 3) Modify mod_autoindex to not use ap_r*. This doesn't solve the underlying problem, but it does remove it from the release. Ryan Index: modules/generators/mod_autoindex.c === RCS file: /home/cvs/httpd-2.0/modules/generators/mod_autoindex.c,v retrieving revision 1.102 diff -u -d -b -w -u -r1.102 mod_autoindex.c --- modules/generators/mod_autoindex.c 20 Mar 2002 17:41:54 - 1.102 +++ modules/generators/mod_autoindex.c 3 Apr 2002 07:12:53 - @@ -980,6 +980,9 @@ * instead of a text document, meaning nothing will be displayed, but * oh well. */ +typedef struct { +apr_bucket_brigade *bb; +} old_write_filter_ctx; static void emit_head(request_rec *r, char *header_fname, int suppress_amble, char *title) { @@ -990,6 +993,7 @@ int emit_H1 = 1; const char *r_accept; const char *r_accept_enc; +old_write_filter_ctx *ctx; /* * If there's a header file, send a subrequest to look for it. If it's @@ -1003,6 +1007,9 @@ if ((header_fname != NULL) && r->args) { header_fname = apr_pstrcat(r->pool, header_fname, "?", r->args, NULL); } + +ctx = apr_pcalloc(r->pool, sizeof(*ctx)); +ap_add_output_filter("OLD_WRITE", ctx, r, r->connection); if ((header_fname != NULL) && (rr = ap_sub_req_lookup_uri(header_fname, r, r->output_filters)) -- Ryan Bloom [EMAIL PROTECTED] 645 Howard St. [EMAIL PROTECTED] San Francisco, CA
RE: 2.0.34 - erratic behavior with autoindexes
> >The core can't do anything about it. If you are creating a sub-request, > >you must flush the data. We could have the core always flush, but that > >would cause us to send flush buckets before any data has been generated, > >or in cases where a flush is not required. The module MUST be the thing > >that does the flush. > > Of course it can... why not simply have ap_run_sub_req automatically flush > the rwrite'n contents of the main request upon invocation, and > automatically > flush the rwritten contents of the subrequest upon completion? > > Then all is transparent That is what I meant by "We could have the core always flush". As I said, that would mean that we are flushing when we don't need too, and we are potentially sending small chunks of data to the socket when we don't need to. It is and should be up to the module to ensure that data is sent to the client when it is supposed to be. Ryan
RE: 2.0.34 - erratic behavior with autoindexes
At 04:27 PM 4/2/2002, you wrote: >The core can't do anything about it. If you are creating a sub-request, >you must flush the data. We could have the core always flush, but that >would cause us to send flush buckets before any data has been generated, >or in cases where a flush is not required. The module MUST be the thing >that does the flush. Of course it can... why not simply have ap_run_sub_req automatically flush the rwrite'n contents of the main request upon invocation, and automatically flush the rwritten contents of the subrequest upon completion? Then all is transparent
RE: 2.0.34 - erratic behavior with autoindexes ( I believe the bug is analyzed in this message)
> > > > We are talking about the main request flushing its > > > > buffers before it runs the request. > > > > > > Um. How'd the buffers get filled before running the request? > > > > The problem is that the main request generates content, and that goes > > into the OLD_WRITE filter buffer. Then it creates a sub-request. If > > the sub-request doesn't use the same OLD_WRITE instance, you have a > > major bug. > > Huh? No... > > Let's say the main req created an OLD_WRITE and put some content in there. > Next, the sub-request inserts its own OLD_WRITE instance (which it won't, > but let's run with this). The subreq then starts jamming content into its > own OLD_WRITE. When the subreq goes away, an EOS will hit the OLD_WRITE in > the sub_req, which will flush the content on up to the main request. The > main request's OLD_WRITE will prepend its data, and continue flushing the > data down the pipe. > > Now, the SUBREQ filter actually intercepts that EOS, but it does pass the > data along. The OLD_WRITEs are emptying their buffers properly. > > Now, the real situation is that a second OLD_WRITE won't be added. When > you > call ap_r*(), it will scan the entire chain looking for an existing > OLD_WRITE to be present. If it is there, then another won't be added. If > the > filter is not at the top, it won't be used. > > And lastly, remember the ordering of the filters. Even if the subreq > inserts > stuff, the OLD_WRITE should always remain at the "top". No. The filters are ordered on a per-request basis, so even if the main request has the OLD_WRITE filter, sub-request filters won't be in filter chain below that OLD_WRITE filter. A graphic might help: If I have a request with the filter chain: OLD_WRITE -> INCLUDES -> byterange -> C-L -> CORE There are two ways to create the sub-request. 1) Point the sub-request at the filter stack, which should produce: (THIS IS WHERE THE BUG IS HAPPENING!!!) SUB_REQ_EOS -> OLD_WRITE -> INCLUDES -> byterange -> C-L -> CORE Any other filters for the sub-request will be inserted before the SUB_REQ_EOS filter. However, what is happening instead, is that filters are being added like: OLD_WRITE -> SUB_REQ_EOS -> INCLUDES -> byterange -> C-L -> CORE The problem with this, is that the sub-request->output_filters pointer is pointing to SUB_REQ_EOS, not to OLD_WRITE, which means that the OLD_WRITE filter is being missed entirely for the sub-request. 2) Don't point the sub-request at the filter stack: (All RESOURCE and CONTENT_SET filters are removed from the sub-request) SUB_REQ_EOS -> byterange -> C-L -> CORE I hope that explains it. I am hoping to look at this in a lot more detail later tonight. Ryan
Re: 2.0.34 - erratic behavior with autoindexes
On Tue, Apr 02, 2002 at 03:30:07PM -0800, Ryan Bloom wrote: >... > > > We are talking about the main request flushing its > > > buffers before it runs the request. > > > > Um. How'd the buffers get filled before running the request? > > The problem is that the main request generates content, and that goes > into the OLD_WRITE filter buffer. Then it creates a sub-request. If > the sub-request doesn't use the same OLD_WRITE instance, you have a > major bug. Huh? No... Let's say the main req created an OLD_WRITE and put some content in there. Next, the sub-request inserts its own OLD_WRITE instance (which it won't, but let's run with this). The subreq then starts jamming content into its own OLD_WRITE. When the subreq goes away, an EOS will hit the OLD_WRITE in the sub_req, which will flush the content on up to the main request. The main request's OLD_WRITE will prepend its data, and continue flushing the data down the pipe. Now, the SUBREQ filter actually intercepts that EOS, but it does pass the data along. The OLD_WRITEs are emptying their buffers properly. Now, the real situation is that a second OLD_WRITE won't be added. When you call ap_r*(), it will scan the entire chain looking for an existing OLD_WRITE to be present. If it is there, then another won't be added. If the filter is not at the top, it won't be used. And lastly, remember the ordering of the filters. Even if the subreq inserts stuff, the OLD_WRITE should always remain at the "top". Cheers, -g -- Greg Stein, http://www.lyra.org/
RE: 2.0.34 - erratic behavior with autoindexes
> > > > The core can't do anything about it. If you are creating a > > sub-request, > > > > you must flush the data. > > > > > > Bing! > > > > > > ANY filter inserted by the subreq could be holding content. The > > subrequest > > > definitely MUST flush out its filter stack before going away. > > > > > > Yes, old_write buffers, and if that is "attached" to the subrequest, > > then > > > it > > > must be flushed. But it really isn't about old_write. > > > > We aren't talking about the sub-request flushing its buffers at the end > > of the request. That happens automatically, because the sub-request > > must send an EOS. > > Right. > > And ignore my comment about the new bucket type for a subreq ending. I > forgot about the EOS thing. > > > We are talking about the main request flushing its > > buffers before it runs the request. > > Um. How'd the buffers get filled before running the request? The problem is that the main request generates content, and that goes into the OLD_WRITE filter buffer. Then it creates a sub-request. If the sub-request doesn't use the same OLD_WRITE instance, you have a major bug. Ryan
Re: 2.0.34 - erratic behavior with autoindexes
On Tue, Apr 02, 2002 at 02:58:11PM -0800, Ryan Bloom wrote: > > > The core can't do anything about it. If you are creating a > sub-request, > > > you must flush the data. > > > > Bing! > > > > ANY filter inserted by the subreq could be holding content. The > subrequest > > definitely MUST flush out its filter stack before going away. > > > > Yes, old_write buffers, and if that is "attached" to the subrequest, > then > > it > > must be flushed. But it really isn't about old_write. > > We aren't talking about the sub-request flushing its buffers at the end > of the request. That happens automatically, because the sub-request > must send an EOS. Right. And ignore my comment about the new bucket type for a subreq ending. I forgot about the EOS thing. > We are talking about the main request flushing its > buffers before it runs the request. Um. How'd the buffers get filled before running the request? Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
On Tue, Apr 02, 2002 at 02:56:33PM -0800, Ryan Bloom wrote: > > Everything below is true, assuming you are not talking about > sub-requests. According to the stack that Greg posted, the OLD_WRITE > filter was not found in the sub-request filter stack. That is to be > expected, because the OLD_WRITE filter is a RESOURCE filter. Since > RESOURCE filters are not kept for sub-requests, the data must be > flushed. Right. > Hm Actually it seems like Resource filters should be kept for > sub-requests that are created with a filter list. If that is the case, > then it looks like the OLD_WRITE filter should be in the filter list. If the sub-request goes, then its resource filters should go. Those filters are associated with that particular sub-request. That implies that the resource filters should be flushed before they die. [ but we should probably have a SUBREQ_ENDING_FLUSH bucket so the SUBREQ filter can chew it up; we don't want to flush all the way to the network just cuz the subreq is disappearing. ] > Unless, the OLD_WRITE filter is being added after the sub-request is > created.. The filter is added at the first call to ap_r*(). We decided on the lazy approach so that pure-brigade systems would not suffer the overhead. Cheers, -g -- Greg Stein, http://www.lyra.org/
Re: 2.0.34 - erratic behavior with autoindexes
Ryan Bloom wrote: > Did you try just http://httpd.apache.org:8092? not until now. Thanks for the extra eyeballs. > It looks like that patch completely broke DirectoryIndex pages. It's definitely hosed. But you and Justin both said something about subrequests which get promoted. That's a new concept to me. I'm not surprised that we would need different logic for promoted and non-promoted subreqs. Greg
RE: 2.0.34 - erratic behavior with autoindexes
> > The core can't do anything about it. If you are creating a sub-request, > > you must flush the data. > > Bing! > > ANY filter inserted by the subreq could be holding content. The subrequest > definitely MUST flush out its filter stack before going away. > > Yes, old_write buffers, and if that is "attached" to the subrequest, then > it > must be flushed. But it really isn't about old_write. We aren't talking about the sub-request flushing its buffers at the end of the request. That happens automatically, because the sub-request must send an EOS. We are talking about the main request flushing its buffers before it runs the request. Ryan
RE: OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
Everything below is true, assuming you are not talking about sub-requests. According to the stack that Greg posted, the OLD_WRITE filter was not found in the sub-request filter stack. That is to be expected, because the OLD_WRITE filter is a RESOURCE filter. Since RESOURCE filters are not kept for sub-requests, the data must be flushed. Hm Actually it seems like Resource filters should be kept for sub-requests that are created with a filter list. If that is the case, then it looks like the OLD_WRITE filter should be in the filter list. Unless, the OLD_WRITE filter is being added after the sub-request is created.. Lots to look at.. Ryan > On Tue, Apr 02, 2002 at 01:50:59PM -0800, Ryan Bloom wrote: > >... > > > chain. Bill S. stuck his head in here and said something about a rule > > > that if old_write is ever used, it has to always be used. > > > > If you read through mod_include, you will see that whenever we create a > > sub-request, we send the output from the current filter to the next > > filter. This is required to make sure that things are output in the > > correct order. If you are a handler and are generating data through > > ap_r* functions, you must flush the filter stack so that you are sure > > that OLD_WRITE isn't buffering for you. > > That's not true. OLD_WRITE was designed to allow a mix/match of writing to > the stack *and* using the ap_r* functions. > > The design is that ap_r* implies a write into the filter stack. That write > is optimized to use a brigade for buffering (so every ap_rputc() doesn't > traverse the filter stack). If OLD_WRITE isn't at the top, then it doesn't > do the buffering (because there are filters between the top and the > OLD_WRITE buffers). > > If somebody writes directly into the filter stack, then it prepends any > buffered content to whatever is written, and then passes it all down the > stack. > > [ just reviewed the code; it seems to still match the original design ] > > It all looks pretty good, but there might be a problem where: > > *) ap_r* is used, so OLD_WRITE is inserted, and the content is stored into >its buffer. > > *) another filter is inserted "above" OLD_WRITE, so it is not the top. > ap_r* >content should now go into this new filter. > > *) ap_r* is called again, but it thinks it cannot buffer. it passes the > new >data directly into the filter stack. > >BUG: the previously-buffered content is not prepended. > > > So... the bug might appear because of the increased dynamicity of the > filter > chain nowadays. Back when OLD_WRITE was written, it was perceived that the > chain would be quite static by the time the first write occurred. > > Note that OLD_WRITE tries quite hard to keep its filter on "top" (its > filter > type is RESOURCE-10). If something broke that, then it is possible to hit > the above bug. > > The fix to the (potential) bug is to change line 1370 in server/protocol.c > to concat the brigades, similar to ap_old_write_filter() (defined just > above, at line 1316). > > Another question is whether anybody defines their filter type as less than > RESOURCE-10 and ends up ahead of OLD_WRITE. > > And lastly: I just realized that if a filter gets in front of OLD_WRITE, > then the branch at line 1366 will get called for each ap_r* call. If that > is > a series of 1000's of ap_rputc() calls, then you'll end up creating > thousands of brigades for those transient buckets(!). > > [ a similar brigade over-creation can occur if you alternate ap_r* and > writing to the filter stack; ap_old_write_filter() will forget about the > brigade it created, so the next ap_r* needs to recreate one ] > > The answer might be to test for brigade-empty rather than ctx->bb == NULL. > Then the branch at 1366 can store and reuse a brigade in ctx->bb. > > Cheers, > -g
Re: 2.0.34 - erratic behavior with autoindexes
On Tue, Apr 02, 2002 at 02:27:17PM -0800, Ryan Bloom wrote: >... > > So it looks like we have to do something about flushing the data that > > old_write is hanging on to. I'd rather see the core figure out how to > > do this, rather than mod_autoindex. Otherwise we are going to impact > > 1.3 modules that use ap_r* more than we need to. Per my previous email, I seriously doubt that old_write is posing a problem here. Yes, it might have buffered content, but per Ryan's point: > The core can't do anything about it. If you are creating a sub-request, > you must flush the data. Bing! ANY filter inserted by the subreq could be holding content. The subrequest definitely MUST flush out its filter stack before going away. Yes, old_write buffers, and if that is "attached" to the subrequest, then it must be flushed. But it really isn't about old_write. > We could have the core always flush, but that > would cause us to send flush buckets before any data has been generated, > or in cases where a flush is not required. The module MUST be the thing > that does the flush. I think the subrequest would need to flush its filter chain before going away. Has nothing to do with the modules, since they don't know they are operating as part of a subreq filter chain. Cheers, -g -- Greg Stein, http://www.lyra.org/
OLD_WRITE stuff (was: Re: 2.0.34 - erratic behavior with autoindexes)
On Tue, Apr 02, 2002 at 01:50:59PM -0800, Ryan Bloom wrote: >... > > chain. Bill S. stuck his head in here and said something about a rule > > that if old_write is ever used, it has to always be used. > > If you read through mod_include, you will see that whenever we create a > sub-request, we send the output from the current filter to the next > filter. This is required to make sure that things are output in the > correct order. If you are a handler and are generating data through > ap_r* functions, you must flush the filter stack so that you are sure > that OLD_WRITE isn't buffering for you. That's not true. OLD_WRITE was designed to allow a mix/match of writing to the stack *and* using the ap_r* functions. The design is that ap_r* implies a write into the filter stack. That write is optimized to use a brigade for buffering (so every ap_rputc() doesn't traverse the filter stack). If OLD_WRITE isn't at the top, then it doesn't do the buffering (because there are filters between the top and the OLD_WRITE buffers). If somebody writes directly into the filter stack, then it prepends any buffered content to whatever is written, and then passes it all down the stack. [ just reviewed the code; it seems to still match the original design ] It all looks pretty good, but there might be a problem where: *) ap_r* is used, so OLD_WRITE is inserted, and the content is stored into its buffer. *) another filter is inserted "above" OLD_WRITE, so it is not the top. ap_r* content should now go into this new filter. *) ap_r* is called again, but it thinks it cannot buffer. it passes the new data directly into the filter stack. BUG: the previously-buffered content is not prepended. So... the bug might appear because of the increased dynamicity of the filter chain nowadays. Back when OLD_WRITE was written, it was perceived that the chain would be quite static by the time the first write occurred. Note that OLD_WRITE tries quite hard to keep its filter on "top" (its filter type is RESOURCE-10). If something broke that, then it is possible to hit the above bug. The fix to the (potential) bug is to change line 1370 in server/protocol.c to concat the brigades, similar to ap_old_write_filter() (defined just above, at line 1316). Another question is whether anybody defines their filter type as less than RESOURCE-10 and ends up ahead of OLD_WRITE. And lastly: I just realized that if a filter gets in front of OLD_WRITE, then the branch at line 1366 will get called for each ap_r* call. If that is a series of 1000's of ap_rputc() calls, then you'll end up creating thousands of brigades for those transient buckets(!). [ a similar brigade over-creation can occur if you alternate ap_r* and writing to the filter stack; ap_old_write_filter() will forget about the brigade it created, so the next ap_r* needs to recreate one ] The answer might be to test for brigade-empty rather than ctx->bb == NULL. Then the branch at 1366 can store and reuse a brigade in ctx->bb. Cheers, -g -- Greg Stein, http://www.lyra.org/
RE: 2.0.34 - erratic behavior with autoindexes
> -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On > Behalf Of Greg Ames > Sent: Tuesday, April 02, 2002 2:35 PM > To: [EMAIL PROTECTED] > Subject: Re: 2.0.34 - erratic behavior with autoindexes > > Greg Ames wrote: > > > > Jeff Trawick wrote: > > > > > > Here's a backtrace while processing /dist/httpd/. Where is the > > > > subrequest filter? > > > > > > maybe this helps... haven't tested yet > > > > > > Index: request.c > > > > yep, this solves about half of the problem. With this on, I now see all > the > > data, it's just in the wrong order. > > I put Jeff's patch on the 2.0.34 test server on daedalus and bounced it. > http://www.apache.org:8092/dist/ looks perfect, since it only has a > HEADER.html. You can see the out-of-order data at > http://www.apache.org:8092/dist/httpd/ (that dir has both a HEADER.html > and > README.html). Did you try just http://httpd.apache.org:8092? It looks like that patch completely broke DirectoryIndex pages. Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Greg Ames wrote: > > Jeff Trawick wrote: > > > > Here's a backtrace while processing /dist/httpd/. Where is the > > > subrequest filter? > > > > maybe this helps... haven't tested yet > > > > Index: request.c > > yep, this solves about half of the problem. With this on, I now see all the > data, it's just in the wrong order. I put Jeff's patch on the 2.0.34 test server on daedalus and bounced it. http://www.apache.org:8092/dist/ looks perfect, since it only has a HEADER.html. You can see the out-of-order data at http://www.apache.org:8092/dist/httpd/ (that dir has both a HEADER.html and README.html). Greg
RE: 2.0.34 - erratic behavior with autoindexes
> Jeff Trawick wrote: > > > > Here's a backtrace while processing /dist/httpd/. Where is the > > > subrequest filter? > > > > maybe this helps... haven't tested yet > > > > Index: request.c > > yep, this solves about half of the problem. With this on, I now see all > the > data, it's just in the wrong order. I see the contents of HEADER.html, > followed > by the contents of README.html, followed by the dirent stuff that > handle_autoindex generates dynamically. The dynamic stuff should be in > the > middle. > > So it looks like we have to do something about flushing the data that > old_write > is hanging on to. I'd rather see the core figure out how to do this, > rather > than mod_autoindex. Otherwise we are going to impact 1.3 modules that use > ap_r* > more than we need to. The core can't do anything about it. If you are creating a sub-request, you must flush the data. We could have the core always flush, but that would cause us to send flush buckets before any data has been generated, or in cases where a flush is not required. The module MUST be the thing that does the flush. As for the patch that did solve the first part of the problem, this is likely to cause problems if the sub-request is promoted to the main request, as is the case with DirectoryIndex. We need to add logic to handle that case. Ryan
Re: 2.0.34 - erratic behavior with autoindexes
Jeff Trawick wrote: > > Here's a backtrace while processing /dist/httpd/. Where is the > > subrequest filter? > > maybe this helps... haven't tested yet > > Index: request.c yep, this solves about half of the problem. With this on, I now see all the data, it's just in the wrong order. I see the contents of HEADER.html, followed by the contents of README.html, followed by the dirent stuff that handle_autoindex generates dynamically. The dynamic stuff should be in the middle. So it looks like we have to do something about flushing the data that old_write is hanging on to. I'd rather see the core figure out how to do this, rather than mod_autoindex. Otherwise we are going to impact 1.3 modules that use ap_r* more than we need to. Greg
RE: 2.0.34 - erratic behavior with autoindexes
> In handle_autoindex, we are working on the main request upon entry. > r->output_filters looks like: > > old_write > byterange > content_length > http_header > core > > When it sees that HEADER.html exists, it apparently fires off a subrequest > (can't say for sure, I hit "n" in gdb rather than "s" at the wrong time :- > ( ). > That finishes, and index_directory starts reading the directory entries > and > creating the html fragments for them. This all seems fine. It writes > them with > ap_rputs, which should buffer the little pieces of data in old_write. > > Once it gets done with the normal files and subdirs, it finds README.html, > and > definitely calls > ap_run_sub_rec for that (I hit "s" this time :-). When we get to > default_handler, r->output_filters looks like: > > includes > byterange > content_length > http_header > core > > The includes filter is using the subrequest's r; byterange, c/l, and > http_header > are using the main request's r; core has a null r. > > If you've been following this list for a while, you probably know that > things > are going to get funky real quick. AFAIK old_write is still buffering all > the > data generated by the main request when the trailer file is sent down the > filter > chain. Bill S. stuck his head in here and said something about a rule > that if > old_write is ever used, it has to always be used. If you read through mod_include, you will see that whenever we create a sub-request, we send the output from the current filter to the next filter. This is required to make sure that things are output in the correct order. If you are a handler and are generating data through ap_r* functions, you must flush the filter stack so that you are sure that OLD_WRITE isn't buffering for you. This will solve SOME of the problem. > * stepping thru the http_header filter on the README/trailer subrequest > was > interesting. It sure looks like that bad boy is generating headers during > the > subrequest. It has a debug assert(!r->main). It doesn't hit because the > r came > from the filter structure, and it's the main request's r. It does take > itself > out of the filter chain after it is done with the trailer subrequest. The http_header filter is never called for the sub-request, because it is only added for the main request. > * what happened to the subrequest filter? what is going to stop > subrequest EOS > buckets from really flushing data to the network prematurely? This is the thing that is most worrying. Without that filter, everything is going to hell in a hand-basket quickly. Ryan
Re: 2.0.34 - erratic behavior with autoindexes
On Tue, Apr 02, 2002 at 04:28:31PM -0500, Jeff Trawick wrote: > maybe this helps... haven't tested yet I believe we took that out specifically to ensure that any requests that may be promoted to a real request do not have the subreq filter. IIRC, if the subreq is being created with a NULL parent request, then it may be "promoted" and will not be served as a subreq. HTH. -- justin
Re: 2.0.34 - erratic behavior with autoindexes
Greg Ames wrote: > > I noticed something weird on daedalus on the test build. It looks like we're > not autoindexing directories with HEADER.html or README.html files correctly. > You see the contents of HEADER.html and/or README.html, but not the other files > or subdirs in the directory. Often, the Mozilla animation keeps going as if the > browser thinks more info is coming. > > Examples of the failure: > > http://www.apache.org:8092/dist/ > http://www.apache.org:8092/dist/httpd/ > http://jakarta.apache.org:8092/builds/ OK, the same bad behavior happens on my ThinkPad with HEAD, so I've been slogging thru it with gdb. I suspect the cause is the change the re-classified the filters and changed when they are added etc. In handle_autoindex, we are working on the main request upon entry. r->output_filters looks like: old_write byterange content_length http_header core When it sees that HEADER.html exists, it apparently fires off a subrequest (can't say for sure, I hit "n" in gdb rather than "s" at the wrong time :-( ). That finishes, and index_directory starts reading the directory entries and creating the html fragments for them. This all seems fine. It writes them with ap_rputs, which should buffer the little pieces of data in old_write. Once it gets done with the normal files and subdirs, it finds README.html, and definitely calls ap_run_sub_rec for that (I hit "s" this time :-). When we get to default_handler, r->output_filters looks like: includes byterange content_length http_header core The includes filter is using the subrequest's r; byterange, c/l, and http_header are using the main request's r; core has a null r. If you've been following this list for a while, you probably know that things are going to get funky real quick. AFAIK old_write is still buffering all the data generated by the main request when the trailer file is sent down the filter chain. Bill S. stuck his head in here and said something about a rule that if old_write is ever used, it has to always be used. Other things that may or may not have a bearing: * the includes filter parses the HEADER and README files, so the subrequest FILE buckets get morphed into MMAP buckets. * stepping thru the http_header filter on the README/trailer subrequest was interesting. It sure looks like that bad boy is generating headers during the subrequest. It has a debug assert(!r->main). It doesn't hit because the r came from the filter structure, and it's the main request's r. It does take itself out of the filter chain after it is done with the trailer subrequest. * what happened to the subrequest filter? what is going to stop subrequest EOS buckets from really flushing data to the network prematurely? Greg
Re: 2.0.34 - erratic behavior with autoindexes
Jeff Trawick <[EMAIL PROTECTED]> writes: > Greg Ames <[EMAIL PROTECTED]> writes: > > > I noticed something weird on daedalus on the test build. It looks like we're > > not autoindexing directories with HEADER.html or README.html files correctly. > > You see the contents of HEADER.html and/or README.html, but not the other files > > or subdirs in the directory. Often, the Mozilla animation keeps going as if the > > browser thinks more info is coming. > > FYI... ripping out INCLUDES doesn't seem to make a difference > (simplification of the problem :) ). > > Here's a backtrace while processing /dist/httpd/. Where is the > subrequest filter? maybe this helps... haven't tested yet Index: request.c === RCS file: /home/cvs/httpd-2.0/server/request.c,v retrieving revision 1.108 diff -u -r1.108 request.c --- request.c 27 Mar 2002 22:42:16 - 1.108 +++ request.c 2 Apr 2002 21:30:33 - @@ -1535,8 +1535,6 @@ rnew->proto_input_filters = r->proto_input_filters; rnew->output_filters = next_filter; rnew->proto_output_filters = r->proto_output_filters; -ap_add_output_filter_handle(ap_subreq_core_filter_handle, -NULL, rnew, rnew->connection); } else { /* If NULL - we are expecting to be internal_fast_redirect'ed @@ -1550,6 +1548,9 @@ rnew->input_filters = r->proto_input_filters; rnew->output_filters = r->proto_output_filters; } + +ap_add_output_filter_handle(ap_subreq_core_filter_handle, +NULL, rnew, rnew->connection); /* no input filters for a subrequest */ -- Jeff Trawick | [EMAIL PROTECTED] Born in Roswell... married an alien...
Re: 2.0.34 - erratic behavior with autoindexes
Greg Ames <[EMAIL PROTECTED]> writes: > I noticed something weird on daedalus on the test build. It looks like we're > not autoindexing directories with HEADER.html or README.html files correctly. > You see the contents of HEADER.html and/or README.html, but not the other files > or subdirs in the directory. Often, the Mozilla animation keeps going as if the > browser thinks more info is coming. FYI... ripping out INCLUDES doesn't seem to make a difference (simplification of the problem :) ). Here's a backtrace while processing /dist/httpd/. Where is the subrequest filter? Why have we made it past ap_content_length_filter()? One result is that the content length is busted. I think the EOS for the subrequest isn't getting eaten and is fouling up things too. (or maybe I forgot *everything* about how filters work :) ) (gdb) where #0 core_output_filter (f=0x812b3d8, b=0x812b5d8) at core.c:3579 #1 0x8070be9 in ap_pass_brigade (next=0x812b3d8, bb=0x81406d0) at util_filter.c:534 #2 0x8060d3e in ap_http_header_filter (f=0x8133808, b=0x81406d0) at http_protocol.c:1472 #3 0x8070be9 in ap_pass_brigade (next=0x8133808, bb=0x81406d0) at util_filter.c:534 #4 0x80731b3 in ap_content_length_filter (f=0x81337f0, b=0x81406d0) at protocol.c:1263 #5 0x8070be9 in ap_pass_brigade (next=0x81337f0, bb=0x81406d0) at util_filter.c:534 #6 0x8062e25 in ap_byterange_filter (f=0x81337d8, bb=0x81406d0) at http_protocol.c:2758 #7 0x8070be9 in ap_pass_brigade (next=0x81337d8, bb=0x81406d0) at util_filter.c:534 #8 0x8077eda in default_handler (r=0x8141048) at core.c:3247 #9 0x8065e37 in ap_run_handler (r=0x8141048) at config.c:193 #10 0x80663d7 in ap_invoke_handler (r=0x8141048) at config.c:373 #11 0x807be70 in ap_run_sub_req (r=0x8141048) at request.c:1867 #12 0x281ebca0 in emit_head (r=0x8133048, header_fname=0x80e4b28 "HEADER", suppress_amble=0, title=0x813eb08 "/dist/httpd") at mod_autoindex.c:1032 #13 0x281edcc4 in index_directory (r=0x8133048, autoindex_conf=0x813d4c0) at mod_autoindex.c:2014 #14 0x281ee006 in handle_autoindex (r=0x8133048) at mod_autoindex.c:2113 #15 0x8065e37 in ap_run_handler (r=0x8133048) at config.c:193 #16 0x80663d7 in ap_invoke_handler (r=0x8133048) at config.c:373 #17 0x80633f3 in ap_process_request (r=0x8133048) at http_request.c:261 #18 0x805e9db in ap_process_http_connection (c=0x812b120) at http_core.c:291 #19 0x806ec3b in ap_run_process_connection (c=0x812b120) at connection.c:85 #20 0x806ef50 in ap_process_connection (c=0x812b120, csd=0x812b048) at connection.c:207 #21 0x8064854 in child_main (child_num_arg=0) at prefork.c:671 #22 0x80649be in make_child (s=0x809dd60, slot=0) at prefork.c:765 #23 0x8064a2b in startup_children (number_to_start=1) at prefork.c:783 #24 0x8064d9a in ap_mpm_run (_pconf=0x809c010, plog=0x80c6010, s=0x809dd60) at prefork.c:998 #25 0x806a825 in main (argc=3, argv=0xbfbffbac) at main.c:618 -- Jeff Trawick | [EMAIL PROTECTED] Born in Roswell... married an alien...
Re: 2.0.34 - erratic behavior with autoindexes
At 04:39 PM 4/1/2002, you wrote: >I noticed something weird on daedalus on the test build. It looks like we're >not autoindexing directories with HEADER.html or README.html files correctly. >You see the contents of HEADER.html and/or README.html, but not the other >files >or subdirs in the directory. Often, the Mozilla animation keeps going as >if the >browser thinks more info is coming. > >Examples of the failure: > >http://www.apache.org:8092/dist/ >http://www.apache.org:8092/dist/httpd/ >http://jakarta.apache.org:8092/builds/ I consider this to be a showstopper to a responsible release of 2.0.34... that release is on -hold-. Let's all try [I'll take a whack in the a.m.] to figure out this pickle. Bill
2.0.34 - erratic behavior with autoindexes
I noticed something weird on daedalus on the test build. It looks like we're not autoindexing directories with HEADER.html or README.html files correctly. You see the contents of HEADER.html and/or README.html, but not the other files or subdirs in the directory. Often, the Mozilla animation keeps going as if the browser thinks more info is coming. Examples of the failure: http://www.apache.org:8092/dist/ http://www.apache.org:8092/dist/httpd/ http://jakarta.apache.org:8092/builds/ The second one gives me different results from IE5 when I hit refresh - sometimes the end of the README.html appears to be missing; sometimes it displays as plain text with the first part of the HEADER.html missing, the http headers visable in the middle, followed by duplicate html. But this one, which doesn't have a HEADER.html or README.html, works fine: http://jakarta.apache.org:8092/builds/jakarta-tomcat/ (get rid of the :8092 to see how the production server behaves.) I have band practice tonight so I'll be offline shortly. I'll leave 2.0.34 up on daedalus in case anyone wants to check it out. Greg