Hi All,
From: Simon Cozens [SMTP:[EMAIL PROTECTED]]
> Bruno Tavares:
>> @lines = <FILE>;
>
>Reading a huge file into memory at once considered harmful.
I know that its harmful. But how can I get perl to stop reading in big data
before its too late? Consider this code which tries to minimse processing
the same input line twice by building a hash.
while (<>) {
$unique{$_} = 1;
if (Total_Data_In_Memory() < 200_000_000 ) {
Process_The_Data(\%unique);
%unique=();
}
}
Process_The_Data(\%unique);
This isn't really FWP I know. But it started off as some fun. I challenged
some .NET c# programmers here to see who could write the fastest
sort-deduplicate program for huge text files (1GB or bigger) that were . The
code I ended up with wasn't the best as I had no idea how much memory perl
had available to it before I had to flush my cache to disk. I could ask the
NT OS how much perl had requested but this number doesn't go down when you
execute %unique=().
Does anyone have a (preferably OS independent) implementation of
"Total_Data_In_Memory"?
Alistair
> ----------------------------------------------------------------------
> Alistair McGlinchy, [EMAIL PROTECTED]
> Sizing and Performance, Central IT, ext. 5012, ph +44 20 7268-5012
> Marks and Spencer, 3 Longwalk Rd, Stockley Park, Uxbridge UB11 1AW, UK
>
-----------------------------------------------------------------------
Registered Office:
Marks & Spencer p.l.c
Michael House, Baker Street,
London, W1U 8EP
Registered No. 214436 in England and Wales.
Telephone (020) 7935 4422
Facsimile (020) 7487 2670
www.marksandspencer.com
Please note that electronic mail may be monitored.
This e-mail is confidential. If you received it by mistake, please let us know and
then delete it from your system; you should not copy, disclose, or distribute its
contents to anyone nor act in reliance on this e-mail, as this is prohibited and may
be unlawful.
The registered office of Marks and Spencer Financial Services Limited, Marks and
Spencer Unit Trust Management Limited, Marks and Spencer Life Assurance Limited and
Marks and Spencer Savings and Investments Limited is Kings Meadow, Chester, CH99 9FB.
