On 04/06/2007 03:27 PM, Bill Stephenson wrote:
Perl crashes when using the included script for large data files. How
can I prevent this from happening?
What do you mean by crashes? Does it abort with an error message placed
in the web-server log file? If so, that's not a crash; it's just an
error abort of the script.
Kindest Regards,
--
Bill Stephenson
Sample data source is a Delimited text file with the following fields:
FMT:STORE:TIKEY:TITLE:QTY:DC:COMPANY:ZONE
DV:043712:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
DV:043712:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
DV:043712:20883070:DVD-AMERICAN PIE 5-NAKE(UR/WS:32.00000:ECDC:Hlyw:4
DV:043712:20883253:DVD-BROOKLYN LOBSTER (WS):2.00000:ECDC:Hlyw:4
DV:043704:20882903:DVD-ALL THE KING'S M('06/WS/2D:16.00000:WCDC:Hlyw:4
DV:043704:20883071:DVD-ALTERED (2006/WS):8.00000:WCDC:Hlyw:4
DV:043693:20882903:DVD-ALL THE KING'S M('06/WS/2D:24.00000:ECDC:Hlyw:4
DV:043693:20883071:DVD-ALTERED (2006/WS):8.00000:ECDC:Hlyw:4
DV:043693:20883070:DVD-AMERICAN PIE 5-NAKE(UR/WS:24.00000:ECDC:Hlyw:4
DV:043693:20883092:DVD-ZOMBIE NATION (2006/WS):4.00000:ECDC:Hlyw:4
DV:043670:20882903:DVD-ALL THE KING'S M('06/WS/2D:8.00000:ECDC:Hlyw:4
<code>
#!/usr/bin/perl
use strict;
use warnings;
# set-up our variable....
my $source = "dataSource.txt";
my $destination = "testOUTPUT.txt";
my $fmt = "";
my $storeno = "";
my $co = "";
my $dc = "";
my $zone = "";
my $store = "";
my $tikey = "";
my $title = "";
my $qty = "";
my $text1 = "";
my $text2 = "";
my $output = "";
my @stores = "";
This creates an array with one element, an empty string. Usually people
just want to create an empty array which can be created like so:
my @stores = ();
# start the work...
open my $IN, $source or die "Can't read source file $source: $!\n";
# put all the stores into an array (@stores)...
my @lines = <$IN>;
foreach my $line(@lines) {
This reads all of the lines into memory at once. Whoa! You almost
certainly want to use a while loop to read the lines one-by-one:
my @lines;
while (my $line = <$IN>) {
Or even better, use Tie::File to treat the lines in the file as an
array. That would almost certainly suit you better since your code uses
an array of lines that you search through more than once.
tie my @lines, 'Tie::File', $source
or die "Can't read file $source: $!\n";
Read the manual pages for Tie::File and perltie:
Start->Run->"perldoc Tie::File"
Start->Run->"perldoc perltie"
($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) =
split(/:/,$line);
push @stores, $store;
}
# remove duplicate stores from the list...
my %seen = ();
my @uniq = ();
foreach my $item (@stores) {
unless ($seen{$item}) {
# if we get here, we have not seen it before
$seen{$item} = 1;
push(@uniq, $item);
}
}
# for each store in our @uniq store list we look for
# lines in our file that have a matching store
foreach my $uniqStore (@uniq) {
foreach my $line(@lines) {
($fmt, $store, $tikey, $title, $qty, $dc, $co, $zone) =
split(/:/,$line);
# if they match we create our entry for the destination file....
if ($store eq $uniqStore) {
$text1= "Street Date $source \n". "Company $co \n".
"Distribution Center $dc \n". "Zone $zone \n". "Store Number $store \n\n";
$text2 .= "$tikey \t $title \t $qty\n";
}
}
# then add it to the final $output.
if ($text1 ne "") {
$output .=
"$text1$text2\n####################\n####################\n\n\f\n";
$text2 = "";
}
}
# and finally print all out
# print $output; #for debugging
open(my $DATA, "> $destination")
or die (" Error 1.3 Couldn't file:: $destination for writing: \n");
print $DATA ($output);
close $DATA;
Once you start using Tie::File, your program will be half its current
size and much faster.
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/
