On December 14, 2001 12:59 pm, Dave Rolsky wrote:
> On Fri, 14 Dec 2001, Perrin Harkins wrote:
> > The thing you were missing is that on an OS with an aggressively caching
> > filesystem (like Linux), frequently read files will end up cached in RAM
> > anyway. The kernel can usually do a better job of managing an efficient
> > cache than your program can.
>
> Plus for some reason IPC overhead seems to seriously degrade as the size
> of the overall shared memory segment increases. I'm not sure why this is
> but I think the docs for IPC::Shareable mention this. Maybe for _very_
> small amounts of data, shared memory might still be a win.
>
Very interesting. We at UF were using IPC caching, in earlier versions of our
comment system. We turfed it for several reasons.
- Locking and storage were taking appreciable wall clock time.
- We had a large memory footprint anyway, and a per-process hash did not
actually add all that much.
- IPC was persistent over server instances, but we already had a dbi
connection, and that is persistent over system reboots as well.
The earlier version was topping out at about 8 hits per second.
So our solution was caching in-process with just a hash, and using a
DBI/mysql persistent store.
in pseudo code
sub get_stuff {
if (! $cache{$whatever} ) {
if !( $cache{whatever} = dbi_lookup()) {
$cache{$whatever}=derive_data_from_original_source($whatever);
dbi_save($cache_whatever);
}
}
$cache{$whatever}
}
Thanks to that and another local content cache implemented with the file
system with time based aging, UF's latest version of the system can do over
100 page views per second on a relatively modest machine (P3 700, 256M IDE,
linux 2.2.x)
The filesystem based / time sensitive aging is a nice little thing we should
put up in CPAN. We've just not done so yet.
--
Jay "yohimbe" Thorne [EMAIL PROTECTED]
Mgr Sys & Tech, Userfriendly.org
