shawn,
thanks for the pointers.
looks like you have saved me from buying more disks ;)!
anjan
On Thu, Aug 21, 2008 at 3:40 PM, Mr. Shawn H. Corey <[EMAIL PROTECTED]>wrote:
> On Thu, 2008-08-21 at 15:23 -0400, ANJAN PURKAYASTHA wrote:
> > ok here is the problem. i have a limited amount of memory.
> >
> > i need to open a large file (~600 megs). i have to open a filehandle,
> read
> > and process each line and write the output result into another file
> (which
> > soon grows to about 300 megs.
> > so,
> >
> > open (IN, "averylargefile");
> > open (OUT, "anotherverylargefile");
> >
> > while (<IN>){
> > chomp;
> > process line here....
> >
> > write OUT ("$abcd\n");
> > }
> >
> > close (IN);
> > close (OUT);
> >
> > Now in order to ensure that all this remains within my quota, i need to
> > delete each line i have read from the input file after i have written my
> > processed output to the output file.
> > this reduction in input file size along with a simultaneous increase in
> the
> > output file size will ensure that i do not overshoot the quota.
> >
> > does anyone have pointers on how to accomplish this in perl.
> >
> > tia,
> > anjan
> >
>
> Go out and buy yourself another disk. Seriously.
>
>
> LOOP:
> Read the first line and process it.
> Read the second line and write were the first was.
> Read the third and write at the end of the second.
> Continue to the end of file.
> Note that if any lines are longer than the previous, you have to read
> more than one to make room to write it.
> Truncate the file after the last write.
> Rewind the file and repeat from LOOP.
>
> See:
> `perldoc -f open` and search for '<+'
> `perldoc -f seek`
> `perldoc -f tell`
> `perldoc -f truncate`
>
>
> --
> Just my 0.00000002 million dollars worth,
> Shawn
>
> "Where there's duct tape, there's hope."
> Cross Time Cafe
>
> "Perl is the duct tape of the Internet."
> Hassan Schroeder, Sun's first webmaster
>
>
--
=============================
anjan purkayastha, phd
bioinformatics analyst
whitehead institute for biomedical research
nine cambridge center
cambridge, ma 02142
purkayas [at] wi [dot] mit [dot] edu
703.740.6939
