also on 64-bit Snow Leopard (Mac OS X 10.6.4)
punk...@lucknow ~$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float,
23171, 23171); print $x->info("%M")."\n"'
perl(85899) malloc: *** mmap(size=18446744071562166272) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!
punk...@lucknow ~$
On Tue, Aug 3, 2010 at 6:36 PM, Benjamin Schuster-Boeckler
<[email protected]> wrote:
> Shockingly, it's true! I just tried the following on a machine with 4 AMD
> Opteron 64bit processors and ~8GB memory:
>
> />$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 23170, 23170); print
> $x->info("%M")."\n"'
> 2047.92Mb
> />$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 23171, 23171); print
> $x->info("%M")."\n"'
> Out of memory!
>
> I chose a matrix of 23170^2 floats because that's 536848900 * 4 bytes, ie
> 1.99GB. Going just a little bit above that limit gives an "Out of memory".
> What's really worrying though is this:
>
> />$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float, 46350, 46350); print
> $x->info("%M")."\n"'
> -8385331.23Kb
>
> Going substantially above the existing wired memory of the machine returns
> interesting negative numbers for the matrix' size, and clearly fails to warn
> about the memory issue. I think we have a bug here?
>
> Ben
>
> On 4 Aug 2010, at 01:10, Chris Marshall wrote:
>
>> On 8/2/2010 9:13 AM, Albrecht I. Schmid wrote:
>>>
>>> since I use perl for years, I like PDL very much. But during the last
>>> months it is apparent that the 2GB limit on pdls is starting to get
>>> prohibitive.
>>
>> I'm confused about the limit on the size of piddles.
>>
>> Could you give an example? I was not aware that there
>> was any limit besides the OS limits (32bit vs 64bit)
>> and maybe the availability of contiguous memory.
>>
>>> Single datasets exceed this limit and reverting to perl loops sort of
>>> removes the performance gain of threading (maybe), besides making coding
>>> ugly, and adding additional (programming) complexity.
>>
>> If there is such a limit, I would consider it a bug
>> (at least on 64bit OSes). If it is not reported, then
>> it cannot be fixed.
>>
>>> Is there a way to enable PDL to handle bigger pdls? If not, I think I
>>> have to move on to Matlab.
>>
>> Good luck there.
>>
>>> Memory is not the issue, btw., 64 bit systems with 10s of GB RAM.
>>
>> I don't have a 64bit OS to work with. Maybe another
>> user with such an OS could comment. BTW, you might
>> want to add the output of your perl -V and the OS
>> information for your platform.
>>
>> Cheers,
>> Chris
>>
>> _______________________________________________
>> Perldl mailing list
>> [email protected]
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
> --
> Benjamin Schuster-Böckler
> Frundsbergstrasse 23a
> 82064 Strasslach
> Deutschland
>
>
>
> _______________________________________________
> Perldl mailing list
> [email protected]
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
-----------------------------------------------------------------------
Assertions are politics; backing up assertions with evidence is science
=======================================================================
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
