Re: LARGE PERL footprint
On Sat, May 20, 2000 at 08:01:36PM +0100, Malcolm Beattie wrote:
Matt Sergeant writes:
On Fri, 19 May 2000, David Larkin wrote:
I require a large array of ints in a real application, just stripped
problem down to bear bones for demo.
Is your array sparse by any chance? If not your modperl daemon is going to
get _much_ larger after you populate that array. If it's sparse, consider
using a hash - that will give you a slight speed penalty at the benefit of
not consuming quite so much ram. Other things to consider:
Re-write this critical bit of code in XS (not as hard as it sounds).
Use a database.
Use the filesystem.
Simply store them in native form in a big long string, just like C
would do. Pre-extend the string first, if you know how big you want it:
$intsize = 4; # alter if sizeof(int) != 4
$intarray = "\0" x (3 * $intsize);
Update index $i with
substr($intarray, $i * $intsize, $intsize) = pack("I", $newval);
which assumes an unsigned int, or use "i" for signed.
To retrieve index $i use
$val = unpack("I", substr($intarray, $i * $intsize, $intsize));
You can even retrive a bunch of them faster with
@ten_to_thirty = unpack("I20", substr(...));
or set a bunch with
substr(..., $intsize * 10) = pack("I20", @ten_to_thirty);
Er, wouldn't using vec() be simpler and faster?
Tim.
Re: LARGE PERL footprint
On Fri, 19 May 2000, David Larkin wrote: I require a large array of ints in a real application, just stripped problem down to bear bones for demo. Is your array sparse by any chance? If not your modperl daemon is going to get _much_ larger after you populate that array. If it's sparse, consider using a hash - that will give you a slight speed penalty at the benefit of not consuming quite so much ram. Other things to consider: Re-write this critical bit of code in XS (not as hard as it sounds). Use a database. Use the filesystem. -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org http://xml.sergeant.org
Re: LARGE PERL footprint
Hi there, On Fri, 19 May 2000, David Larkin wrote: Can anyone help explain why PERL gives such a large memory footprint advise how to get around it. In addition to the other suggestions, you might want to try use integer; in the bits of your Perl code that manipulate integers. I guess I'm paying the price for PERL not being strongly typed, a feature I really like ( until now ;-) ) Hmmm. I require a large array of ints in a real application Er, real (as opposed to floating point:) ? 73, Ged.
Re: LARGE PERL footprint
[EMAIL PROTECTED] (G.W. Haywood) wrote: Hi there, On Fri, 19 May 2000, David Larkin wrote: Can anyone help explain why PERL gives such a large memory footprint advise how to get around it. My general philosophy (well, at least in these matters) is that large chunks of reference data should be stored outside the application, in a database or other appropriate place. Let the code be the code, and the data be the data. If that's not possible, you can check out some of the CPAN modules that let you store data as bare C data. I've never had the need to use one, so I can't recommend anything. I require a large array of ints in a real application Er, real (as opposed to floating point:) ? As opposed to imaginary, probably. Or complex. =) ------ Ken Williams Last Bastion of Euclidity [EMAIL PROTECTED]The Math Forum
Re: LARGE PERL footprint
Matt Sergeant writes:
On Fri, 19 May 2000, David Larkin wrote:
I require a large array of ints in a real application, just stripped
problem down to bear bones for demo.
Is your array sparse by any chance? If not your modperl daemon is going to
get _much_ larger after you populate that array. If it's sparse, consider
using a hash - that will give you a slight speed penalty at the benefit of
not consuming quite so much ram. Other things to consider:
Re-write this critical bit of code in XS (not as hard as it sounds).
Use a database.
Use the filesystem.
Simply store them in native form in a big long string, just like C
would do. Pre-extend the string first, if you know how big you want it:
$intsize = 4; # alter if sizeof(int) != 4
$intarray = "\0" x (3 * $intsize);
Update index $i with
substr($intarray, $i * $intsize, $intsize) = pack("I", $newval);
which assumes an unsigned int, or use "i" for signed.
To retrieve index $i use
$val = unpack("I", substr($intarray, $i * $intsize, $intsize));
You can even retrive a bunch of them faster with
@ten_to_thirty = unpack("I20", substr(...));
or set a bunch with
substr(..., $intsize * 10) = pack("I20", @ten_to_thirty);
For that last bit of extra performance, do
sub INTSIZE () { 4 }; # alter if sizeof(int) != 4
--Malcolm
--
Malcolm Beattie [EMAIL PROTECTED]
Unix Systems Programmer
Oxford University Computing Services
Re: LARGE PERL footprint
On Fri, 19 May 2000, David Larkin wrote:
Can anyone help explain why PERL gives such a large memory
footprint advise how to get around it.
Running the simple script below, I get a footprint of 63 MB
about 22 bytes per int.
The C program only 11748 K ... 4 bytes per int
#!/usr/local/bin/perl
for ( $i=0 ; $i 300 ; $i++ )
{
$X[$i]=int(1);
}
main()
{
int x[300];
sleep(60);
}
I guess I'm paying the price for PERL not being strongly typed,
a feature I really like ( until now ;-) )
I require a large array of ints in a real application, just stripped
problem down to bear bones for demo.
I'd be grateful for any advice
When you really need the C/C++ slimness use XS to glue C/C++ code for your
Perl code, don't just switch to C, which can be not development speed
wise.
A great XS tutorial (especially for simple things like your example):
http://perlmonth.com/columns/modules/modules.html?issue=6
http://perlmonth.com/columns/modules/modules.html?issue=7
http://perlmonth.com/columns/modules/modules.html?issue=8
http://perlmonth.com/columns/modules/modules.html?issue=9
http://perlmonth.com/columns/modules/modules.html?issue=10
Enjoy! And encourage Steven McDougall (the author) to write more of this
great stuff :)
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: LARGE PERL footprint
On Fri, 19 May 2000, David Larkin wrote: Can anyone help explain why PERL gives such a large memory footprint advise how to get around it. Your array might be smaller if you pre-extend it to the size you need (see perldata). You could also look at some of the sneaky bit vector modules on CPAN if your data is appropriate for this. And you could try switching between perl's malloc and your system's malloc, though this requires you to build a new version of perl. - Perrin
