On Thu, May 22, 2014 at 9:27 AM, Yossi Itzkovich
<[email protected]> wrote:
> Hi,
>
> Here is a small test that shows the problem (memory consumption of counting
> the size of a huge hash, with not-so-small keys)
>
> -------
> my %hash;
> for (my $i=0; $i<5000000;$i++){
> $hash{$i x 100}=$i};
> print "OK\n";
> sleep 20; ### Here the memory is about 3G
> print (scalar keys %hash,"\n");
> sleep 10 ### here the memory is 3.5 G
> ------
>
> Well, it means the list is being built, and it does consume a lot of memory
> (and CPU- but this is not my problem here). In my real application it
> crashes, since I have bigger hash/keys, and I am getting "out of memory"
> crash.
>
> Isn't there a better way to get the hash size ?
I ran a few tests and I think there is some problem with that measurement.
Has your system started to swap for this tests to run?
Try to run the same tests at lower numbers where your system does not
use the swap space yet.
For me, when I ran it with smaller numbers the memory increase was 0.
In any case if the application crashes due to lack of memory, you will
probably need to increase the memory.
Reaching the swap will kill the performance even if it does not crash
the script yet.
Finally, you could run 'each' on the hash and count yourself, but
after I ran my measurements on
I seriously doubt the "scalar keys %hash" uses more than a few bytes of memory.
here is the script I used:
use strict;
use warnings;
use 5.010;
use Memory::Stats;
my $stats = Memory::Stats->new;
$stats->start;
$stats->checkpoint("before my big method");
my %hash;
for (my $i=0; $i<4000000;$i++){
$hash{$i x 100}=$i
};
print "OK\n";
$stats->checkpoint("after hash created");
for (0 .. 9) {
say "$_: ", scalar keys %hash;
}
$stats->checkpoint("after hash counter");
$stats->stop;
$stats->report;
_______________________________________________
Perl mailing list
[email protected]
http://mail.perl.org.il/mailman/listinfo/perl
