Hey David thanks for the reply
so the brackets are a bad idea here ???
the file that I'm loading has compressed information,
so I preprocessed a file that has compressed information
sort of like a zip file, except the format used is the
BER Integer compression format so the compression
data file is purely numerical,
and the lookup table holds addresses
of each chunk of the table in the ram,
so each chunk is a different size
this code here is not intended to describe how I compress the information
it just describes vaguely what format it is in and thats the BER compressed
integer format
so while none of the 3 gigabytes of information is in the ram; using this
process
I can accessall the information at any point in the file with no wait time,
based on the line number in a file...
-Mark
________________________________
From: David Mertens <[email protected]>
To: MARK BAKER <[email protected]>
Cc: perldl <[email protected]>
Sent: Wednesday, February 15, 2012 3:37 AM
Subject: Re: [Perldl] How to find out cause of out of memory
Mark (and anybody else following along) -
First, to your idea. Your code creates a line lookup table for a text file.
This isn't data compression and you're not loading all 3 gigabytes of data into
40 megabytes of ram. You're simply creating a lookup table that is 40 megabytes
which allows you to quickly locate the position of the data you need. The
system still has to go back to the disk to fetch the data with
# I'm using $fh instead of DATA here:
seek $fh, $offset[$entered], 0;
$line = <$fh>;
Second, to your code. This code works, but not for the reasons you might
suspect. Try running your code under "use strict" and you will see errors. Why?
Every time you surround code with a block you create a lexical scope. Any "my"
variables within that scope are not accessible outside that scope (PadWalker
shenanigans aside). As such, lines like
{my $line_num = 2;}
do the following: (1) creates a scoped block, (2) creates a lexical variable
within that scoped block called , (3) assigns the value of 2 to that lexical
variable, and (4) exits the scoped block, destroying all lexicals in the
process. Your code works because you are essentially using global variables
instead of lexical variables without realizing it and Perl's default
interpretation of undef in numeric context evaluates to zero. In more modern
and idiomatic Perl, your code might look like this:
-------------------%<-------------------
use strict;
use warnings;
# Get the file:
print "What file would you like to examine? ";
chomp (my $file_name = <>);
die "File does not exist!\n" unless -f $file_name;
open my $fh, '<', $file_name or die "Unable to open $file_name\n";
# Build lookup table (recall: $. is the current record number)
my @offset = (0);
$offset[$.] = tell $fh while (<$fh>);
# Retrieve data for the user:
while (1) {
# Get the requested line number
print "Give me a line offset between 0 and $#offset (or q to quit): ";
chomp (my $line = <>);
# Exit if that's what they want
exit if lc $line eq 'q';
# Verify it:
if ($line =~ /^\d+$/ and $line < @offset) {
# Go to the lookup position and read the data:
seek $fh, $offset[$line], 0;
my $data = <$fh>;
# No need to chomp; we need the \n anyway:
print $data;
}
else {
print "Bad input, try again\n";
}
}
------------------->%-------------------
the trick here is to put your file data into decimal
>
>then to pack the file data like this
>
>##################################################
>
>$data = pack "w*", $_;
>#################################################
>then just use unpack to view the numerical data
>
What are you doing here? I've never seen the 'w' pack template, but perldoc
tells me it is "a BER compressed integer." And you're only packing a single
number, whatever happens to be in "it" at the moment.
David
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl