On 08 Sep 2014, at 8:53 PM, Ruediger Pluem <rpl...@apache.org> wrote:
> Wouldn't it make more sense instead of using an empty brigade to create yet > another metabucket that signals write > completion? It could also contain information how much data to send down the > chain for single filters if they e.g. send > heap or transient buckets. Otherwise how should they know? > If you have a filter that has a large file bucket set aside and it does > transform it e.g. to a heap bucket during it's > processing because it changes data on it I guess it doesn't make sense if it > does send all stuff once it gets triggered > for write completion as we would end up in a blocking write then in the core > filter. But if it knows how much is left in > the core filter buffer it could try to just sent this and avoid thus blocking > writes. And if there is no room left in > the buffer or if what is left is too small for the filter to operate on it, > the filter could just pass the bucket down > the chain and if it would end up in the core output filter, the core output > filter would just try to write what it has > buffered. I spent a lot of time going down this path of having a dedicated metabucket, and quickly got bogged down in complexity. The key problem was “what does a filter actually do when you get one”, it was unclear and it made my head bleed. That makes life hard for module authors and that is bad. As I recall there were also broken filters out there that only knew about FLUSH and EOS buckets (eg ap_http_chunk_filter()). The problem we’re trying to solve is one of starvation - no filters can set aside data for later (except core via the NULL hack), because there is no guarantee that they’ll ever be called again later. You have to write it now, or potentially write it never. The start of the solution is ensure filters aren’t starved: if you have data in the output filters - and obviously you have no idea which filters have setaside data - you need a way to wake them all up. The simplest and least disruptive way is to pass them all an empty brigade, job done. We’ve got precedent for this - we’ve been sending NULL to the core filter to achieve the same thing, we want something that works with any filter. The second part of the problem is filters biting off more than they can chew. Example: give mod_ssl a 1GB file bucket and mod_ssl won’t yield until that entire 1GB file has been sent for the reason (now solved) above. The next step to enable write completion is to teach filters like mod_ssl to yield when handling large quantities of data. The core filter has an algorithm to yield, including various checks for flow control and sanity with respect to file handles. If a variant of this algorithm could be exposed generically and made available to critical filters like mod_ssl, we’ll crack write completion. Regards, Graham —
