Alan Raetz wrote:
So I have a perl application that upon startup loads about ten perl hashes (some of them complex) from files. This takes up a few GB of memory and about 5 minutes. It then iterates through some cases and reads from (never writes) these perl hashes. To process all our cases, it takes about 3 hours (millions of cases). We would like to speed up this process. I am thinking this is an ideal application of mod_perl because it would allow multiple processes but share memory.
Sure you could use modperl for this. I would also consider at least these alternatives: - use Cache::FastMmap, https://metacpan.org/pod/Cache::FastMmap Load up your data with a loader script, and forget about it. Cache::FastMmap also works with modperl. - use a network server, like memcached or redis to store your read-only data, and use a lightweight network protocol (on localhost) to get the data. In both cases, reading from multiple processes will not be a problem. The cheapest solution for the consumer part (the "cases" above) would be to use a command like "parallel" to fire up as many copies of your consumer script as you can afford. Hope this helps, -- Cosimo
