On Sun, Apr 28, 2002 at 05:18:24PM +0200, Ernest Lergon wrote:
> Hi,
>
> in a mod_perl package I load a CSV file on apache startup into a simple
> hash as read-only class data to be shared by all childs.
>
> A loading routine reads the file line by line and uses one numeric field
> as hash entry (error checks etc. omitted):
>
> package Data;
>
> my $class_data = {};
>
> ReadFile ( 'data.txt', $class_data, 4 );
>
> sub ReadFile
> {
> my $filename = shift; # path string
> my $data = shift; # ref to hash
> my $ndx_field = shift; # field number
>
> my ( @record, $num_key );
> local $_;
>
> open ( INFILE, "<$filename" );
> while ( <INFILE> )
> {
> chomp;
> @record = split "\t";
> $num_key = $record[$ndx_field];
> $data->{$num_key} = [ @record ];
> }
> close ( INFILE );
> }
>
> sub new...
> creates an object for searching the data, last result, errors etc.
>
> sub find...
> method with something like:
> if exists $class_data->{$key} return...
> etc.
>
> Now I'm scared about the memory consumption:
>
> The CSV file has 14.000 records with 18 fields and a size of 2 MB
> (approx. 150 Bytes per record).
>
> Omitting the loading, top shows, that each httpd instance has 10 MB (all
> shared as it should be).
>
> But reading in the file explodes the memory to 36 MB (ok, shared as
> well)!
>
> So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
> hash?
>
> Reading perldebguts.pod I did not expect such an increasing:
>
> Description (avg.) CSV Perl
> 4 string fields (4 chars) 16 bytes (32 bytes) 128 bytes
> 9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
> 5 string fields (rest) 89 bytes (32 bytes) 160 bytes
> the integer key (20 bytes) 20 bytes
> 150 bytes 524 bytes
>
> That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
I tried this program in Perl (outside of modperl) and the memory
consumption is only 4.5MB:
#!/usr/bin/perl -w
$foo = {};
for ($i = 0; $i < 14000; $i++) {
$foo->{sprintf('%020d', $i)} = 'A'x150;
}
<>;
1;
So I suggest something else might be going on causing your memory
problems.
-jwb
