Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an error...."File too large" and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
Q2. Is there a way I can parallely do all these things? i.e
downloading.....unzipping....data extraction... database operations.
Here is the script.. if somebody can help me optimize it.
########################################################################
####
@filenames=("A.gz","B.gz","....", "...",.............);
open(ZP,"database_zipped_archive.dat");
while (<@filenames>)
{
[EMAIL PROTECTED];
$ftp->get("$ftpfile");
$unzippedftpfile="unzipped.txt";
open IN,"gzip -d < $ftpfile >$unzippedftpfile |";
close(IN);
$subst=substr($_,0,2);
open(ZNP,"tempfile.txt") or die "tempfilenot ready.....f: $!\n";;
while (<ZNP>)
{
if ($subst=~/XXX/)
{
Some Operations .....................
push(@XXX,xxxxxxx);
}
if ($subst=~/YYY/)
{
Some Operations .....................
push(@YYY,yyyyy);
}
.
.
.
.
.
}
$filenumber++;
}
my $th = $db->prepare("INSERT INTO XXX_Table VALUES
(?,?)");
[EMAIL PROTECTED];
while (<@XXX>)
{
while ($checkorg < $len)
{
$th->bind_param(1, $checkorg);
$th->bind_param(2, @XXX->[$checkorg]);
my $rc = $th->execute || die "Can't execute statement:
$DBI::errstr";
$checkorg++;
}
}
$checkorg=0;
my $th = $db->prepare("INSERT INTO YYY_Table VALUES
(?,?)");
[EMAIL PROTECTED];
while (<@YYY>)
{
while ($checkorg < $len)
{
$th->bind_param(1, $checkorg);
$th->bind_param(2, @YYY->[$checkorg]);
my $rc = $th->execute || die "Can't execute statement:
$DBI::errstr";
$checkorg++;
}
}
.
.
.
.
.
################################################
Thanks In advance
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
