Cache::FastMmap is a great module for sharing read/write data, but it can't compete with the speed of loading it all into memory before forking as Alan said he plans to do.
- Perrin On Tue, Feb 3, 2015 at 2:05 AM, Cosimo Streppone <cos...@streppone.it> wrote: > Alan Raetz wrote: > > So I have a perl application that upon startup loads about ten perl >> hashes (some of them complex) from files. This takes up a few GB of >> memory and about 5 minutes. It then iterates through some cases and >> reads from (never writes) these perl hashes. To process all our cases, >> it takes about 3 hours (millions of cases). We would like to speed up >> this process. I am thinking this is an ideal application of mod_perl >> because it would allow multiple processes but share memory. >> > > Sure you could use modperl for this. > I would also consider at least these alternatives: > > - use Cache::FastMmap, https://metacpan.org/pod/Cache::FastMmap > Load up your data with a loader script, and forget about it. > Cache::FastMmap also works with modperl. > > - use a network server, like memcached or redis to store your > read-only data, and use a lightweight network protocol (on localhost) > to get the data. > > In both cases, reading from multiple processes will not be a problem. > The cheapest solution for the consumer part (the "cases" above) > would be to use a command like "parallel" to fire up as many copies > of your consumer script as you can afford. > > Hope this helps, > > -- > Cosimo > >
