Re: File size problem
Vasudev.K. wrote:
Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an errorFile too large and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
This sounds like a [rpblem with your sip application, or perhaps the
limitatins of the format.
Q2. Is there a way I can parallely do all these things? i.e
downloading.unzippingdata extraction... database operations.
Here is the script.. if somebody can help me optimize it.
I'm not sure about optimixations at this point, because there are some
formatting issues that make it unnecessarily difficult to follow the code.
Herewith some tips on indentation:
use strict;
use warnings;
@filenames=(A.gz,B.gz,, ...,.);
#lose this, nd all other meaningless blank lines
#use vlank lines only when they signal some meaningful transition in the
flow of executtion.
open(ZP,database_zipped_archive.dat);
while (@filenames)
{
All code within this block should be indented by however many spaces you are
using as your standard. Use spaces, not tabs.
[EMAIL PROTECTED];
$ftp-get($ftpfile);
$unzippedftpfile=unzipped.txt;
open IN,gzip -d $ftpfile $unzippedftpfile |;
close(IN);
$subst=substr($_,0,2);
open(ZNP,tempfile.txt) or die tempfilenot ready.f: $!\n;;
while (ZNP) # This line should not be indented
The control statement of the while loop is in the main flow of execution, as
is the opening brace.
Anyway, I am sorry that I cannot take the time to read this code for
content. Please clean up the indentation so that it is not random, but
rather reflects the logic of program execution. Then re-post the code.
Here are the basics:
Choose a number of space characters as a standard indent increm,ent. Taste
vary, but choose a standard and stick with it.
Any time you have an opening brace for a set of lines, this should be a
signal to indent the following line by one increment.
Unindent by one increment on the line that has the closing brace. The
closing brace should always have a line to itself.
Here is a sample usng a 4-space indent. [and one space for line
continuance]. Some people prefer up to eight spaces. I personally use
two. Just be consistent with whatever you choose.
sub add_interior_children {
my $self = shift;
my ($roots, $class_system, $resolved,
$unresolved, $to_close) = @_;
while (my $key = shift @{$unresolved}) {
my $parent = $class_system-get_parent($key);
next unless $parent;
if ($resolved-{$parent}) {
$self-add_interior_child($key, $parent,
$class_system, $resolved, $to_close);
} else {
push @{$unresolved}, $key;
}
}
}
When you can look at the code from arms length and tell, without reading it,
how the general flow of execution goes, repost it, and you should get plenty
of help with the logic.
Joseph
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
File size problem
Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an errorFile too large and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
Q2. Is there a way I can parallely do all these things? i.e
downloading.unzippingdata extraction... database operations.
Here is the script.. if somebody can help me optimize it.
@filenames=(A.gz,B.gz,, ...,.);
open(ZP,database_zipped_archive.dat);
while (@filenames)
{
[EMAIL PROTECTED];
$ftp-get($ftpfile);
$unzippedftpfile=unzipped.txt;
open IN,gzip -d $ftpfile $unzippedftpfile |;
close(IN);
$subst=substr($_,0,2);
open(ZNP,tempfile.txt) or die tempfilenot ready.f: $!\n;;
while (ZNP)
{
if ($subst=~/XXX/)
{
Some Operations .
push(@XXX,xxx);
}
if ($subst=~/YYY/)
{
Some Operations .
push(@YYY,y);
}
.
.
.
.
.
}
$filenumber++;
}
my $th = $db-prepare(INSERT INTO XXX_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@XXX)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @XXX-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
$checkorg=0;
my $th = $db-prepare(INSERT INTO YYY_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@YYY)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @YYY-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
.
.
.
.
.
Thanks In advance
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: File size problem
-Original Message- Vasudev.K. wrote: . Q1. After unzipping, the file is huge (even the zipped one is :(( ).. almost 5GB. The system throws an errorFile too large and exits. How do I get around this ache? One way I want to do it is unzipped file into many parts and process each part separately but cant get to write a code for it. Is there any better solution? . Try hjsplit.exe to split a huge file into many pieces. It is a freeware product (available on Internet) that can manage file of 10GB and over. It doesn't need installation, it is a simple executable ... Then download each piece separately... E. LOQUENDO S.p.A. Vocal Technology and Services [EMAIL PROTECTED] CONFIDENTIALITY NOTICE This message and its attachments are addressed solely to the persons above and may contain confidential information. If you have received the message in error, be informed that any use of the content hereof is prohibited. Please return it immediately to the sender and delete the message. Should you have any questions, please contact us by replying to [EMAIL PROTECTED] Thank you -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: File size problem
Ya.. I guess .. I got a part of the answer..
I am unzipping it onto the STDOUT and reading it from there
But... still stuck with parallel processing :p:D
-Original Message-
From: Vasudev.K. [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 26, 2003 1:06 PM
To: [EMAIL PROTECTED]
Subject: File size problem
Hi,
I have this rather critical problem, I am trying to download quite huge
files from a remote server through ftp. (The file being in a zipped
format). I have an array which stores the names of the files to be
downloaded. I am opening each of them up at my end and extracting data
out of it and storing it in my oracle database.
Q1. After unzipping, the file is huge (even the zipped one is :(( )..
almost 5GB. The system throws an errorFile too large and exits.
How do I get around this ache? One way I want to do it is unzipped file
into many parts and process each part separately but cant get to write a
code for it. Is there any better solution?
Q2. Is there a way I can parallely do all these things? i.e
downloading.unzippingdata extraction... database operations.
Here is the script.. if somebody can help me optimize it.
@filenames=(A.gz,B.gz,, ...,.);
open(ZP,database_zipped_archive.dat);
while (@filenames)
{
[EMAIL PROTECTED];
$ftp-get($ftpfile);
$unzippedftpfile=unzipped.txt;
open IN,gzip -d $ftpfile $unzippedftpfile |;
close(IN);
$subst=substr($_,0,2);
open(ZNP,tempfile.txt) or die tempfilenot ready.f: $!\n;;
while (ZNP)
{
if ($subst=~/XXX/)
{
Some Operations .
push(@XXX,xxx);
}
if ($subst=~/YYY/)
{
Some Operations .
push(@YYY,y);
}
.
.
.
.
.
}
$filenumber++;
}
my $th = $db-prepare(INSERT INTO XXX_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@XXX)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @XXX-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
$checkorg=0;
my $th = $db-prepare(INSERT INTO YYY_Table VALUES
(?,?));
[EMAIL PROTECTED];
while (@YYY)
{
while ($checkorg $len)
{
$th-bind_param(1, $checkorg);
$th-bind_param(2, @YYY-[$checkorg]);
my $rc = $th-execute || die Can't execute statement:
$DBI::errstr;
$checkorg++;
}
}
.
.
.
.
.
Thanks In advance
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
