piping the split.
Hello, I want to unzip a file and run another perl script on it. So I tried to pipe the STDOUT to the script -- system(gzip -d abc.ndx | perl myscript.pl); but I suppose the STDOUT is huge so half way the system gives an error... file too large and exits. So I tried to split the STDOUT and run the perl script.. -- system(gzip -d abc.ndx | split -1000 | perl myscript.pl); But that doesn't solve the purpose as I need to run the script on each split. Can anybody suggest a better way of doing it? Thanks in advance -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
File size problem
Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an errorFile too large and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
Q2. Is there a way I can parallely do all these things? i.e
downloading.unzippingdata extraction... database operations.
Here is the script.. if somebody can help me optimize it.
@filenames=(A.gz,B.gz,, ...,.);
open(ZP,database_zipped_archive.dat);
while (@filenames)
{
[EMAIL PROTECTED];
$ftp-get($ftpfile);
$unzippedftpfile=unzipped.txt;
open IN,gzip -d $ftpfile $unzippedftpfile |;
close(IN);
$subst=substr($_,0,2);
open(ZNP,tempfile.txt) or die tempfilenot ready.f: $!\n;;
while (ZNP)
{
if ($subst=~/XXX/)
{
Some Operations .
push(@XXX,xxx);
}
if ($subst=~/YYY/)
{
Some Operations .
push(@YYY,y);
}
.
.
.
.
.
}
$filenumber++;
}
my $th = $db-prepare(INSERT INTO XXX_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@XXX)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @XXX-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
$checkorg=0;
my $th = $db-prepare(INSERT INTO YYY_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@YYY)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @YYY-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
.
.
.
.
.
Thanks In advance
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: File size problem
Ya.. I guess .. I got a part of the answer..
I am unzipping it onto the STDOUT and reading it from there
But... still stuck with parallel processing :p:D
-Original Message-
From: Vasudev.K. [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 26, 2003 1:06 PM
To: [EMAIL PROTECTED]
Subject: File size problem
Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an errorFile too large and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
Q2. Is there a way I can parallely do all these things? i.e
downloading.unzippingdata extraction... database operations.
Here is the script.. if somebody can help me optimize it.
@filenames=(A.gz,B.gz,, ...,.);
open(ZP,database_zipped_archive.dat);
while (@filenames)
{
[EMAIL PROTECTED];
$ftp-get($ftpfile);
$unzippedftpfile=unzipped.txt;
open IN,gzip -d $ftpfile $unzippedftpfile |;
close(IN);
$subst=substr($_,0,2);
open(ZNP,tempfile.txt) or die tempfilenot ready.f: $!\n;;
while (ZNP)
{
if ($subst=~/XXX/)
{
Some Operations .
push(@XXX,xxx);
}
if ($subst=~/YYY/)
{
Some Operations .
push(@YYY,y);
}
.
.
.
.
.
}
$filenumber++;
}
my $th = $db-prepare(INSERT INTO XXX_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@XXX)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @XXX-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
$checkorg=0;
my $th = $db-prepare(INSERT INTO YYY_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@YYY)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @YYY-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
.
.
.
.
.
Thanks In advance
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
