On Fri, 3 Sep 2010, Graham Leggett wrote:
<snip>
Regarding the issue of the disk cache cramming the entire file into
memory/address space, an alternate solution could be that the cache returns
buckets pointing to the cached file, ie that the cache consumed those pesky
mmapped buckets. This way the cache could cache the file rather quickly
independent of the speed of the client, so that the file caching is
finished in a sane time so others can benefit from it even though the
cache-initiating request is running at snail-speed...
Been keen to do this for a while, this would definitely solve the RAM
problem, but wouldn't solve the time problem. Copying 4GB of data from a slow
disk can easily take minutes, and when Blu-ray images start becoming common,
the problem would get worse.
Yup. The next step to solve that would be to be able to serve requests
from cache while they are being cached. I don't know the RFC
implications of doing this, but in real life it's really useful.
For those who have forgotten, that's what we do in our
large-file-caching-patchset for mod_disk_cache (hidden as an
attachment to https://issues.apache.org/bugzilla/show_bug.cgi?id=39380
but I should really get around to upload an up2date version that
applies cleanly to the current 2.2 release). Some of the solutions
there aren't really applicable to httpd proper (mostly workarounds for
missing infrastructure), but some ideas are rather sane (like writing
the header files in a single go with an iovec with null terminated
strings instead of crlf-stuff thad needs to be parsed). Oh, and the
design caters for a shared data cache (ftp and rsync access uses the
same cache), which isn't really a priority for something in httpd
proper.
Sadly I haven't had any time to whack this into pieces and feed the
appropriate parts into httpd trunk, but I can at least vouch for the
fact that the functionality is really useful and essential when
dealing with large files (ie dvd/bluray images) and that development
in this direction is desired.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se
---------------------------------------------------------------------------
Must Go - Some Jehovahs witnesses need shouting at.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
