Re: Reading and from more than one filehandle / memory question
On Wednesday, November 5, 2003, at 05:20 PM, [EMAIL PROTECTED]
wrote:
This is my first attempt. My misunderstanding was in the fact that I
could
put the filehandles inside another filehandles while loop.
Good start. Now I'll help. ;)
#!perl
#!/usr/bin/perl
...is more standard.
use strict;
and...
use warnings;
read_the_files;
Yuck. Don't do this. Call subs with:
sub_name();
Trust me.
While we're at it, don't stick in a sub just for the sake of a sub. If
you're defining and calling it at the same place, what's the point?
sub read_the_files{
print \nread_the_files;#
my$file1 =./file1.txt;
#contents of file1.txt
#1
#6
my$file2 = ./file2.txt;
#contents of file2.txt
#5
#10
my$file3 = ./file3.txt;
#contents of file3.txt
#;;;4;
#;;;9;
my$file4 = ./file4.txt;
#contents of file3.txt
I believe this is a misprint and you meant #...file4.txt
#5
#10
This is identical to file2.txt. What do you do then, overwrite?
my$writing = ./combined.txt;
my$count;
unless(open(FH, $file1)){print \nCouldn't read
$file1;exit}else{print
\nOpen successful}
###
#this next bit of code seems old and sloppy.
[ snip rest of code }
You're right, sloppy. I mean that in the best possible way though. :D
Anytime you find yourself typing nearly the same lines over and over
again, you're probably missing a needed loop. That's exactly what
loops are for in fact, repeating code.
Here's my first crack at your problem, see what you think:
#!/usr/bin/perl
use strict;
use warnings;
my @results;
for my $file (qw(file1 file2 file3 file4)) {
open IN, '', $file.txt or die Input error: $!\n;
while (IN) {
$results[$. - 1] = [ ] unless $results[$. - 1];
$results[$. - 1][length $1] = $2 if /^(;*)(\d+)/;
}
close IN;
}
open OUT, '', 'combined.txt' or die Output error: $!\n;
{
no warnings 'uninitialized';
print OUT join(':', @$_), \n foreach @results;
}
close OUT;
Even better, in my opinion, would be to remove those hard coded files
and make a UNIX filter you would call with something like:
perl filter file?.txt combined.txt
That looks like this:
#!/usr/bin/perl
use strict;
use warnings;
my @results;
while () {
$results[$. - 1] = [ ] unless $results[$. - 1];
$results[$. - 1][length $1] = $2 if /^(;*)(\d+)/;
close ARGV if eof;
}
{
no warnings 'uninitialized';
print join(':', @$_), \n foreach @results;
}
Well, hopefully that at least gives you a couple of ideas. Good luck.
James
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Reading and from more than one filehandle / memory question
On Thu, 06 Nov 2003 10:31:18 -0600, James Edward Gray II wrote: my @results; [...] Wasn't it the whole point that the OP _couldn't_ store this in an array, as it would consume too much memory? The solution goes like this; 1. Define the files in an array 2. Start reading the first file 3. For each line in the first file, read a line from the other files. Push the lines to an array. 4. Process the array, output the result, go back to #3. -- Tore Aursand [EMAIL PROTECTED] -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Reading and from more than one filehandle / memory question
On Thursday, November 6, 2003, at 11:13 AM, Tore Aursand wrote:
On Thu, 06 Nov 2003 10:31:18 -0600, James Edward Gray II wrote:
my @results;
[...]
Wasn't it the whole point that the OP _couldn't_ store this in an
array,
as it would consume too much memory? The solution goes like this;
You are correct, of course. I had forgotten this and as the test data
he posted wasn't anywhere near a problem, I opted for the simple way
out. Thanks for keeping me honest.
James
#!/usr/bin/perl
use strict;
use warnings;
my @files;
my $index = 0;
for my $file (qw(file1 file2 file3 file4)) {
open $files[$index++], '', $file.txt or die Input error: $!\n;
}
open OUT, '', 'combined.txt' or die Output error: $!\n;
do {
my @line;
foreach my $file (@files) {
my $line = $file;
$line[length $1] = $2 if defined($line) $line =~ /^(;*)(\d+)/;
}
{
no warnings 'uninitialized';
print OUT join(':', @line), \n;
}
} until (eof $files[0]);
close $_ foreach @files;
close OUT;
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Reading and from more than one filehandle / memory question
I have some large files that all have data for filling one table. I'm concerned about memory resources. 1. I want to read from each of these files one line at a time at the same time without assigning the file to an array. I haven't done this one yet, Does anyone have any example code? I want to pass the lines in an array to a subroutine which does # 2. 2. Then combine the lines, plugging in elements according to index if that index hasn't been filled. This one is pretty easy to figure out.
Re: Reading and from more than one filehandle / memory question
On Wednesday, November 5, 2003, at 01:47 PM, [EMAIL PROTECTED] wrote: I have some large files that all have data for filling one table. I'm concerned about memory resources. 1. I want to read from each of these files one line at a time at the same time without assigning the file to an array. I haven't done this one yet, Does anyone have any example code? I want to pass the lines in an array to a subroutine which does # 2. There's no real logic problem here, that I can see. You should go about it just the way you said. Open both files. Read a line from the first file and a line from the second, sticking them in separate variables, of course. Then make your subroutine call... I can post some code, if you need me to, but that's pretty straight forward stuff. Want to give it a go first and yell if you get stuck? I think you'll surprise yourself. James 2. Then combine the lines, plugging in elements according to index if that index hasn't been filled. This one is pretty easy to figure out. -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Reading and from more than one filehandle / memory question
Once again, please say who you are? [EMAIL PROTECTED] wrote: I have some large files that all have data for filling one table. I'm concerned about memory resources. 1. I want to read from each of these files one line at a time at the same time without assigning the file to an array. I haven't done this one yet, Does anyone have any example code? I want to pass the lines in an array to a subroutine which does # 2. A non-starter! If you don't read the file into an array than you can't pass that array to your subroutine! 2. Then combine the lines, plugging in elements according to index if that index hasn't been filled. This one is pretty easy to figure out. I can start to guess what you mean, but I could go a long way down the wrong path if I tried. Tie::File may help, but please explain more about what your files contain and how you want to correlate them? It would also help if you could say something about the size of your files compared with the amount of virtual memory that you have. Rob -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Reading and from more than one filehandle / memory question
On Wed, 05 Nov 2003 14:47:53 -0500, Motherofperls wrote: I have some large files that all have data for filling one table. What is a table in this context? 1. I want to read from each of these files one line at a time at the same time without assigning the file to an array. Easy enough; Just open both files and read one line at a time from both of them. But what do you want to do if one of the files have more lines in it than the other? Does anyone have any example code? For what? Reading from files? What are you having trouble with? -- Tore Aursand [EMAIL PROTECTED] -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Reading and from more than one filehandle / memory question
This is my first attempt. My misunderstanding was in the fact that I could
put the filehandles inside another filehandles while loop.
#!perl
use strict;
read_the_files;
sub read_the_files{
print \nread_the_files;#
my$file1 =./file1.txt;
#contents of file1.txt
#1
#6
my$file2 = ./file2.txt;
#contents of file2.txt
#5
#10
my$file3 = ./file3.txt;
#contents of file3.txt
#;;;4;
#;;;9;
my$file4 = ./file4.txt;
#contents of file3.txt
#5
#10
my$writing = ./combined.txt;
my$count;
unless(open(FH, $file1)){print \nCouldn't read $file1;exit}else{print
\nOpen successful}
###
#this next bit of code seems old and sloppy. I would have to do it on all
the files to get the last line number
# is there a newer way?
my$test = FH;
while( FH){$count = $.;}
close(FH);
print \ncount: .$count;
###
unless(open(FH1, $file1)){print \nCouldn't read $file1;exit}else{print
\nOpen successful}
unless(open(FH2, $file2)){print \nCouldn't read $file2;exit}
unless(open(FH3, $file3)){print \nCouldn't read $file3;exit}
unless(open(FH4, $file4)){print \nCouldn't read $file4;exit}
unless(open(FHW, $writing)){print \nCouldn't write to $writing;exit;}
#get the info for each line and put it into the database line by line this
will use less memory
while($count--){
print \n reading filehandle;
my($line1,$line2,$line3,$line4);
$line1 = FH1;print \nline1: $line1;
$line2 = FH2;print \nline2: $line2;
$line3 = FH3;print \nline3: $line3;
$line4 = FH4;print \nline4: $line4;
#putting content in empty spaces so full aray will register
$line1 =~ s/;;/;`;/g;
$line2 =~ s/;;/;`;/g;
$line3 =~ s/;;/;`;/g;
$line4 =~ s/;;/;`;/g;
#the sizes of these arrays may vary by one so I have to find the greatest one
[EMAIL PROTECTED] = split(;, $line1);
[EMAIL PROTECTED] = split(;, $line2);
[EMAIL PROTECTED] = split(;, $line3);
[EMAIL PROTECTED] = split(;, $line4);
my$n1 = @insert1;
my$n2 = @insert2;
my$n3 = @insert3;
my$n4 = @insert4;
[EMAIL PROTECTED] = [$n1,$n2,$n3,$n4];
print \n ns : $n1,$n2,$n3,$n4;
[EMAIL PROTECTED] = sort { $b = $a } @ns;
[EMAIL PROTECTED];
for(my$i=0;$i7;$i++){
if($insert1[$i] =~ /\w/){$mainline[$i] = $insert1[$i]}
if($insert2[$i] =~ /\w/){$mainline[$i] = $insert2[$i]}
if($insert3[$i] =~ /\w/){$mainline[$i] = $insert3[$i]}
if($insert4[$i] =~ /\w/){$mainline[$i] = $insert4[$i]}
}
my$printline = join(;, @mainline);
#remove backtics that had no values
$printline =~ s/`//g;
print FHW $printline.\n;
}
}
#add to mysql table after FWH is updated
Re: Reading and from more than one filehandle / memory question
[EMAIL PROTECTED] wrote:
This is my first attempt. My misunderstanding was in the fact that I could
put the filehandles inside another filehandles while loop.
#!perl
use strict;
read_the_files;
sub read_the_files{
print \nread_the_files;#
my$file1 =./file1.txt;
#contents of file1.txt
#1
#6
my$file2 = ./file2.txt;
#contents of file2.txt
#5
#10
my$file3 = ./file3.txt;
#contents of file3.txt
#;;;4;
#;;;9;
my$file4 = ./file4.txt;
#contents of file3.txt
#5
#10
Hold it! You're doing all this work to somehow combine a total of eight lines of
text? This does not make sense. Please post some real data and an
English-language explanation of how the contents of the files relate to each
other. Is it a line-by-line correlation? What are all the serial semicolons
for? Are they empty field delimiters? If so, there is no good reason to inject
anything into them. Perl can handle null strings just fine when splitting. I
try to be kind to myself though, by following delimiters with a space in
text-based data files. perl can also handle a little well-placed white space
quite easily.
Think your problem our in words first. Articulate it carefully. The code can
come almost effortlessly once the logic is worked out.
Joseph
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
