Brian Akins wrote:
So this does not work in reverse proxy situation? Those buckets are not file-based.
It should do - the network input filter isn't going to hand buckets up the filter chain that don't fit in RAM, so special handling isn't required. A normal read bucket / write to cache file works for this, most of my testing used an URL served via a reverse proxy (quick and dirty way to slow down the network enough to see what was going on).
What I have done in our proxy is to use "spool" files for large responses. Basically, write the incoming data from origin to a temp file, then use file buckets to serve this to client. Not ideal, but it seems to work for us for 300MB objects. The downside to this is that for requests being cached, you are writing to disk twice for each "caching" request (proxy spool and cache temp). However, this trade-off is better than proxy eating all the RAM.
So far my gut feel on this is that disk_cache really should not be worrying about whether bucket reads are going to crash anything, disk_cache shouldn't be doing bucket reads at all.
If the bucket was passed to a write_to_file output filter, the special bucket handling could be hidden in there, and disk_cache need never know.
First of all, if you use deterministic temp file names, you allow only one thread to attempt to cache an object at a time. This helps on popular, "mid-sized" objects when they are being cached.
As I understand it, with the patches committed so far, deterministic file names are being used. Or at least, downloading an URL from the cache a few hundred times with a concurrency of 50 using ab, finished very shortly after the lead thread (triggered by Firefox) had finished downloading the original slow reverse proxied file.
Regards, Graham --
