Re: Streaming compression of output from mod_perl handler?
Sorry again. I've just checked sources and found that if mod_deflate received flush then it flushes both zlib and Apache. You can try to set autoflush in perl module with $|=1; or call $r-rflush; when you like to flush output. I just tried using $r-rflush in my handler and it works perfectly :-) The output gets rendered on the fly and it barely hurts the compression ratio. The 430 KB gets compressed to 26 KB instead of 24.5 KB :-) :-) But wouldn't it be nice to have some mod_deflate option where you could specify that mod_deflate should flush and send the currently compressed output every time it had received a certain amount of input or every time it had generated a certain amount of output. We are, for instance, using a template module to generate the output. We just give the template module a data structure and a template, and it then goes away and fills in the template, outputting HTML. This means that we don't have any easy way of calling flush at certain So we don't have any easy way of calling rflush once in a while. Of course we might just modify or substitute the template module, but it seems like a more automatic kind of streaming deflating (like described above) would be nice to have. Nicholas Oxhøj
Re: Streaming compression of output from mod_perl handler?
What mod_deflate did you try ? My or Apache 2.0 ? Yours I can comment my mod_deflate. First, mod_deflate did not collect all output before compressing - it compress it on the fly. But it emits compressed content in 8K block. It's Apache's HUGE_STRING_LEN #define and it can be changed in sources. Besides if some module flushes output then mod_deflate would flushed it too. mod_deflate did not have problems with chunked transfer-encoding because it compress content before Apache start to make chunks. mod_deflate remove Content-Length header so compressed content would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). At the moment I have made my mod_perl handler extra slow, so that it is easier to see the effects of the different compression modules. The handler returns approx. 430KB of uncompressed HTML. With no compression module, the HTML slowly trickles to the browser, and the data gets displayed/rendered as it arrives. With mod_deflate enabled, nothing gets displayed in the browser until all 24KB of the gzip compressed HTML has arrived at the browser. Unfortunately I don't really have the tools to see when data are received at the client side, I can only judge by the time they are displayed/rendered in the browser. So I can't really tell if the problem is, that all the data are received in one big batch or if they are actually received in 8K blocks, but some bug-like feature in IE 5.5 makes it unable to decode the gzipped data in a streaming fashion as they arrive. Nicholas Oxhøj
Re: Streaming compression of output from mod_perl handler?
if mod_deflate will receive flush request it will flush deflate encoding and will write compressed content to Apache buffer. But it does not flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS usually should send data if in OS's buffer there is more then 2K. So it's probably MSIE feature as well as NC4 can not render tables until it receive '/table'. If it was a bug in MSIE, it must be something specifically related to receiving compressed content, since the same data sent uncompressed, gets rendered as they arrive. Anyway, I just tried getting the same data using lynx, and this made it evident that *without* mod_deflate, the data gets sent by Apache as they are ready, whereas *with* mod_deflate, all the compressed data are sent as one big block at the end. So it seems that I am still unable to get the functionality I am looking for. Nicholas Oxhøj
Re: Streaming compression of output from mod_perl handler?
I've developed an Apache::Dynagzip handler for Outlook Technologies, Inc. to serve the dynamic content with the option to control the size of the chunk(s). It works fine (as standing along, and within the Apache::Filter Chain). Using the Apache::Dynagzip you have several options to control your chunk size(s). You can even control the size of every chunk from the source generator: Just send the mask of the end of the chunk to the Apache::Dynagzip within the outgoing content/stream. (You might wish to send your header to the client browser while creating the rest of the body...) Otherwise, it is buffering the outgoing stream up to the length of the chunk's minimum size , which is not less then declared (2K default). It certainly sounds very promising. Can I find the Apache::Dynagzip handler anywhere - Google returns nothing... Is your handler Apache::Filter Chain compatible? Not at the moment. But as it is only a content producer, isn't the only requirement, that I insert $r-filter_register() somewhere near the top of the handler, before returning any output? At least that is what I did, when I tried to test the Apache::Compress handler (which I didn't succeed in getting to work properly). Nicholas Oxhøj
Re: Streaming compression of output from mod_perl handler?
I'm not sure that lynx can handle compressed response on the fly - it uses gzip in pipe. The best way to test it using netcat. Well, lynx didn't decompress it, it just output the gzip compressed content to stdout. As I didn't have netcat readily available on the machine, I instead put an strace on lynx, to be absolutely sure, that it didn't receive any output until the very end - and it didn't :-( I you like to test I can make patch for mod_deflate to flush Apache. But if major browsers can not handle compressed content on the fly it's not valuable. That would be an interesting patch, but with approx 450KB of uncompressed HTML, I would expect mod_deflate to receive compressible input, regardless if the content producer specifically flushes or not. But I might have misunderstood something. Regarding the browsers ability to handle compressed content on the fly, we probably won't know until I find a module that is able to produce such output. Nicholas Oxhøj
Streaming compression of output from mod_perl handler?
Hi I am looking for an Apache module which will allow me to compress the output of my mod_perl handler (a native handler, i.e. not running under Apache::Registry). But since my handler can potentially take a long time to finish, the output has to be compressed in a streaming fashion (or in small blocks) so that the browser will start receiving data before my handler has completely finished. I have been experimenting with all the different Apache compression modules I have been able to find, but have not been able to get the desired result. I have tried Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with different results. One I cannot get to work at all. Most work, but seem to collect all the output before compressing it and sending it to the browser. There also seems to be an issue about the new HTTP/1.1 chunked transfer-encoding. For instance, mod_gzip will not compress chunked output, unless you allow it to dechunk it by collecting all the output and compressing it as one big block. So I am basically looking for anyone who has had any success in achieving this kind of streaming compression, who could direct me at an appropriate Apache module. Regards, Nicholas Oxhøj
Apache::Registry caching of compiled scripts
Hi Does anybody know why Apache::Registry caches compiled scripts by their URI, instead of by their absolute path? At my site we have quite a lot of virtual hosts, which use the same base of cgi scripts. This causes the same scripts to be compiled and cached several times (one time for each virtual host). As I understand it, this behaviour COULD be changed by putting $Apache::Registry::NameWithVirtualHost = 0 in a perl start-up file, which would only cache on the relative URI (without the host name), but this won't really do, since not ALL the virtual hostst use the same base of CGI scripts. So I was just wondering why the compiled scripts are not cached under the absolute path to the script, instead of under the URI, as the former seems quite logical at first and would seem to solve some performance/memory issues when using virtual hosts. This design choice probably has a very good reason, which I am just unable to see. I would just like to find out in order to learn... :-) Nicholas Oxhøj
