Re: Too little memory

[Camilo Gonzalez]

Bob Showalter wrote:

[redirected to [EMAIL PROTECTED]

Camilo Gonzalez wrote:
 

I need to read in from a temp file that is about 10 megs big in 1.5
meg increments and write results to a database. I can't slurp up the
whole temp file because I'm only allowed 2 megs of memory. I was
hoping to read in only 1.5 megs per pass of the file but the read
seems to be slurping up the whole thing. Do I need sysread?
Anyway, here's my attempt:

open (TEMP, "dataTemp.txt") or die "Can't open dataTemp for
reading:$!"; my $offset = 0;
while (read TEMP, my $tempbuf, 1500000, $offset)
   

Because you increment $offset by 1500000 at the end of the loop, you're
extending $tempbuf with each pass through the loop.
Okay, let me make sure I understand read correctly. The arguments it 
takes is filehandle, scalar, length and offset. I always thought offset 
is where it will begin reading the filehandle in bytes, i.e. if I 
specify an offset of 1500000, it will begin reading TEMP at that point 
and continue for lengh or another 1500000. Am I wrong? If so, how do I 
read in a specified length? Remember, I only have 2 megs of memory to 
work with. How can I flush out everything so I get a fresh 2 megs with 
every pass?

A hint: When you write big numbers in perl, you can use underscores to make
them more readable:
  $size = 1_500_000;      # less likely to add or drop a zero!

If you don't use $offset, you'll fix that problem. However, you'll introduce
another, because the logical record that spans the 1.5 Mb boundary won't be
processed correctly. You'll have to add logic to handle that.
Is the input data not delimited by a known string that you could set $/ to?
Are input rows variously delimited by \r, \n, and \f?
I'm working on a Macwith OS X and it tends to mangle and randomly(?) add 
line feeds.

 

      {
my @temp = split/[\r\n\f]/, $tempbuf;
foreach my $temp(@temp)
          {
                  my ($label,
                  $genre,
                  $catalogue_num,
                  $upc,
                  $artist,
                  $title,
                  $description,
                  $format,
                  $shop_price,
                  $dist_price,
                  $dist_1,
                  $dist_2,
                  $vendor,
                  $cost,
                  $release_date,
                  $special_buy,
                  $items_buy_unit,
                  $items_sell_unit,
                  $quant_break,
                  $asset_acct,
                  $buy,
                  $buy_unit_measure,
                  $expense_cos_account,
                  $inactive_item,
                  $income_acct,
                  $inventory,
                  $item_picture,
                  $minimum_level,
                  $selling_base_price,
                  $price_level_b,
                  $price_level_c,
                  $price_level_e,
                  $price_level_f,
                  $sell,
                  $sell_unit_measure,
                  $quantity_break) = split/\t/, $temp;
next if $genre eq "";

my $inserts = "INSERT INTO bigdata VALUES (
                  '$label',
                  '$genre',
                  '$catalogue_num',
                  '$upc',
                  '$artist',
                  '$title',
                  '$description',
                  '$format',
                  '$shop_price',
                  '$dist_price',
                  '$dist_1',
                  '$dist_2',
                  '$vendor',
                  '$cost',
                  '$release_date',
                  '$special_buy',
                  '$items_buy_unit',
                  '$items_sell_unit',
                  '$quant_break',
                  '$asset_acct',
                  '$buy',
                  '$buy_unit_measure',
                  '$expense_cos_account',
                  '$inactive_item',
                  '$income_acct',
                  '$inventory',
                  '$item_picture',
                  '$minimum_level',
                  '$selling_base_price',
                  '$price_level_b',
                  '$price_level_c',
                  '$price_level_e',
                  '$price_level_f',
                  '$sell',
                  '$sell_unit_measure',
                  '$quantity_break')";
my $sth = $dbh->do($inserts) or warn "Can't load <TEMP> into bigdata:
$!:$@"; 
   

If the database driver you're using allows placeholders, you can simplify
this greatly. Also, using a prepared statement (if supported) can speed this
up as well.
 

Placeholders allowed but prepare staements are not.

 



       }
       $offset += 1500000;
       }
close (TEMP);
   

 



--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]