Hi Richard,
On Friday 01 October 2010 05:33:20 Richard Green wrote:
> Hi everyone, could use some advice on a perl script I wrote using hashes. I
> have three files ( each file is a list of indexes) my program loads these
> indexes into hashes and compares the differences and similarities between
> them. With smaller files it runs fine. problem is I now files have about 88
> million records and the script has been running for days. not sure the best
> way to resolve the issue.(pieces of code samples below) One suggestion that
> was given to me was to load the first file into a hash and then as I open
> the next file I immediately do a comparison of the second hash into the 1st
> one for similarities and differences. I can't really get my head around how
> to that. Is there a simple way to compare three very large hashes without
> so much demand on memory? The program manages to loads the large hashes
> without problems and as I mentioned previously smaller files have no
> issues. Any suggestions folks have are muchly appreciated.
> Thanks R
>
Well, your code seems fine except for the fact that you used the
«$$my_hash_ref{$key}» notation instead of the clearer «$my_hash_ref->{$key}»
(which shouldn't matter to perl). There may be some CPAN modules to do what
you're doing more efficiently. However, I should note that on most systems
perl will have a difficulty managing a hash with 88 million records, because
perl variables and data structure have a lot of overhead.
Just as an anecdote, I recently worked on a C program which manages a gigantic
hash in memory and I could only manage about 20 million records before it
exceeded the 3 GB RAM in my x86-64 Linux laptop and reached the swap. (It's a
development branch of Freecell Solver - http://fc-solve.berlios.de/ if you're
interested.)
As a result, I strongly recommend that you instead delegate the hard work to a
database:
http://perl-begin.org/uses/databases/
Regards,
Shlomi Fish
--
-----------------------------------------------------------------
Shlomi Fish http://www.shlomifish.org/
Why I Love Perl - http://shlom.in/joy-of-perl
<rindolf> She's a hot chick. But she smokes.
<go|dfish> She can smoke as long as she's smokin'.
Please reply to list if it's a mailing list post - http://shlom.in/reply .
--
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
http://learn.perl.org/
