Shockingly, it's true! I just tried the following on a machine with 4 AMD
Opteron 64bit processors and ~8GB memory:
/>$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 23170, 23170); print
$x->info("%M")."\n"'
2047.92Mb
/>$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 23171, 23171); print
$x->info("%M")."\n"'
Out of memory!
I chose a matrix of 23170^2 floats because that's 536848900 * 4 bytes, ie
1.99GB. Going just a little bit above that limit gives an "Out of memory".
What's really worrying though is this:
/>$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 46350, 46350); print
$x->info("%M")."\n"'
-8385331.23Kb
Going substantially above the existing wired memory of the machine returns
interesting negative numbers for the matrix' size, and clearly fails to warn
about the memory issue. I think we have a bug here?
Ben
On 4 Aug 2010, at 01:10, Chris Marshall wrote:
> On 8/2/2010 9:13 AM, Albrecht I. Schmid wrote:
>>
>> since I use perl for years, I like PDL very much. But during the last
>> months it is apparent that the 2GB limit on pdls is starting to get
>> prohibitive.
>
> I'm confused about the limit on the size of piddles.
>
> Could you give an example? I was not aware that there
> was any limit besides the OS limits (32bit vs 64bit)
> and maybe the availability of contiguous memory.
>
>> Single datasets exceed this limit and reverting to perl loops sort of
>> removes the performance gain of threading (maybe), besides making coding
>> ugly, and adding additional (programming) complexity.
>
> If there is such a limit, I would consider it a bug
> (at least on 64bit OSes). If it is not reported, then
> it cannot be fixed.
>
>> Is there a way to enable PDL to handle bigger pdls? If not, I think I
>> have to move on to Matlab.
>
> Good luck there.
>
>> Memory is not the issue, btw., 64 bit systems with 10s of GB RAM.
>
> I don't have a 64bit OS to work with. Maybe another
> user with such an OS could comment. BTW, you might
> want to add the output of your perl -V and the OS
> information for your platform.
>
> Cheers,
> Chris
>
> _______________________________________________
> Perldl mailing list
> [email protected]
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
--
Benjamin Schuster-Böckler
Frundsbergstrasse 23a
82064 Strasslach
Deutschland
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
