On 8/4/2010 6:22 AM, Ingo Schmid wrote:
>
> first of all, thanks for the many replies. I was not aware that this
> issue was unknown. I can try debugging the issue, I have access to
> enough memory, but little to no knowledge of perls internals.
>
>
> I ran the following test:
> for $i (0..2**27) { $str.='abcdefghijklmnopqrstuvwxyz0123456789';},
> that's a bit more than 2**32 (4GB). Took a few seconds to run.
> Then
> perldl> p length ($str)
> 4831838244
>
> perldl> p length ($str)/1024/1024/1024
> 4.50000003352761
>
> So I conclude it is not an underlying perl/string limitation, correct?
> Ingo
Thanks for running the check. It appears confirmed that
this is a limitation of the current PDL allocation routines
that call the perl api SvGROW() but with a size as an
int rather than STRLEN type. That puts the limit at 2**31-1
for piddle sizes.
The fix will be to change the usage of the allocation to
use the proper STRLEN type. Unfortunately, it is intimately
related to the working of PDL at the lowest level so the
change may break things elsewhere that use the piddles.
It might be possible to have a shorter term fix with the
int type replaced by unsigned int to push the limit to
4GB per piddle. Since it is the same word length, that
could improve things in the short term.
However, we're in the final stages of the pre PDL-2.4.7
release process so this might have to wait until after
August to be looked at in more detail. In the meantime,
I'll open a feature request on sf.net for the support
of larger piddles.
Cheers,
Chris
> PS: My machine is unstable gentoo ~amd64, we have ubuntu boxes also.
>
> uname -a
>
> Linux spectre 2.6.33-gentoo-r2 #4 SMP Thu Jul 29 12:26:35 CEST 2010
> x86_64 Intel(R) Xeon(R) CPU W3520 @ 2.67GHz GenuineIntel GNU/Linux
>
> 12GB RAM
>
> perl -V:
>
>
> Summary of my perl5 (revision 5 version 12 subversion 1) configuration:
>
> Platform:
> osname=linux, osvers=2.6.33-gentoo, archname=x86_64-linux
> uname='linux spectre 2.6.33-gentoo #2 smp tue apr 6 10:24:11 cest
> 2010 x86_64 intel(r) xeon(r) cpu w3520 @ 2.67ghz genuineintel gnulinux '
> config_args='-des -Duseshrplib -Darchname=x86_64-linux
> -Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O2 -pipe -march=core2
> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
> -Dprefix=/usr -Dsiteprefix=/usr -Dvendorprefix=/usr
> -Dprivlib=/usr/lib64/perl5/5.12.1
> -Darchlib=/usr/lib64/perl5/5.12.1/x86_64-linux
> -Dsitelib=/usr/lib64/perl5/site_perl/5.12.1
> -Dsitearch=/usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
> -Dvendorlib=/usr/lib64/perl5/vendor_perl/5.12.1
> -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
> -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
> -Dsiteman1dir=/usr/share/man/man1 -Dsiteman3dir=/usr/share/man/man3
> -Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3
> -Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.12.1 -Dlocincpth=
> -Duselargefiles -Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost
> -dperladmin=r...@localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm
> -Di_ndbm -Di_gdbm -Di_db -Dinc_version_list=5.12.0 5.12.0/x86_64-linux
> -Dusrinc=/usr/include -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64'
> hint=recommended, useposix=true, d_sigaction=define
> useithreads=undef, usemultiplicity=undef
> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
> use64bitint=define, use64bitall=define, uselongdouble=undef
> usemymalloc=n, bincompat5005=undef
> Compiler:
> cc='x86_64-pc-linux-gnu-gcc', ccflags ='-fno-strict-aliasing -pipe
> -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
> optimize='-O2 -pipe -march=core2 -fomit-frame-pointer -msse4
> -msse4.1 -msse4.2 -mcx16 -msahf',
> cppflags='-fno-strict-aliasing -pipe -fstack-protector'
> ccversion='', gccversion='4.4.4', gccosandvers=''
> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize=8
> alignbytes=8, prototype=define
> Linker and Libraries:
> ld='x86_64-pc-linux-gnu-gcc', ldflags =' -fstack-protector'
> libpth=/usr/local/lib64 /lib64 /usr/lib64
> libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
> libc=/lib/libc-2.11.2.so, so=so, useshrplib=true,
> libperl=libperl.so.5.12.1
> gnulibc_version='2.11.2'
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
> cccdlflags='-fPIC', lddlflags='-shared -O2 -pipe -march=core2
> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
> -fstack-protector'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
> USE_64_BIT_ALL
> USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO
> USE_PERL_ATOF
> Locally applied patches:
> 0001-gentoo_MakeMaker-RUNPATH.diff
> 0002-gentoo_config__over.diff
> 0003-gentoo_cpan__definstalldirs.diff
> 0004-gentoo_cpanplus__definstalldirs.diff
> 0005-gentoo_create-libperl-soname.diff
> 0006-gentoo_MakeMaker-delete__packlist.diff
> 0007-fixes_8d66b3f9__h2hp__fix.diff
> 0008-fixes_ef9df645__glob__crashes__when__File__Glob__is__empty.diff
>
> 0009-fixes_e3d01d03__Naif__calls__segfault__T__PRTOBJ__of__the__stock__typemap.diff
> Built under linux
> Compiled at Jul 26 2010 11:18:49
> @INC:
> /usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
> /usr/lib64/perl5/site_perl/5.12.1
> /usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
> /usr/lib64/perl5/vendor_perl/5.12.1
> /usr/lib64/perl5/5.12.1/x86_64-linux
> /usr/lib64/perl5/5.12.1
> /usr/lib64/perl5/site_perl
> /usr/lib64/perl5/vendor_perl
>
> On 08/04/2010 03:59 AM, Chris Marshall wrote:
>> On 8/3/2010 9:43 PM, Christian Soeller wrote:
>>> Is it possible to change things so that 64 bit sizes
>> > can be passed in the two places you identified and see
>> > if that works?
>>> I appreciate that things could still fall over in various
>> > PP autogenerated code pieces if ints are used for offset
>> > calculations in slice and other vaffine operations.
>>
>> It could work. The problem is it needs someone with
>> a 64bit OS, lots of memory, and a willingness to
>> debug the issue. I don't have any *large* memory
>> systems at the moment. Not that I wouldn't like to
>> have one. :-)
>>
>> --Chris
>>
>>
>>> On 4/08/2010, at 12:46 PM, Chris Marshall wrote:
>>>
>>>> I took a further look at the SvGROW calls in
>>>> PDL/Basic/Core routines and the two that I found
>>>> both use int type for their sizes. That would
>>>> limit a piddle size to<2**31 or about 2GB.
>>>>
>>>> It looks like 64bit support for PDL may need
>>>> to be added to the list for the future. I don't
>>>> know the scope of the changes that would be
>>>> required to support larger PDL data objects.
>>>>
>>>> --Chris
>>>>
>>>> On 8/3/2010 8:39 PM, Chris Marshall wrote:
>>>>> On 8/3/2010 8:30 PM, P Kishor wrote:
>>>>>> On Tue, Aug 3, 2010 at 7:19 PM, Chris Marshall<[email protected]>
>>>>>> wrote:
>>>>>>> On 8/3/2010 8:01 PM, P Kishor wrote:
>>>>>>>> also on 64-bit Snow Leopard (Mac OS X 10.6.4)
>>>>>>>>
>>>>>>>> punk...@lucknow ~$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float,
>>>>>>>> 23171, 23171); print $x->info("%M")."\n"'
>>>>>>>> perl(85899) malloc: *** mmap(size=18446744071562166272) failed (error
>>>>>>>> code=12)
>>>>>>>> *** error: can't allocate region
>>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>>> Out of memory!
>>>>>>>> punk...@lucknow ~$
>>>>>>> What is perl -V?
>>>>> I looked at the PDL/Basic/Core stuff and it looks like
>>>>> if SvGROW can handle a>2GB string then, in principle,
>>>>> PDL should be able to handle piddles of that size.
>>>>>
>>>>> Could you see if you can create a string more than 2GB
>>>>> long? It might take a while but it will tell us if the
>>>>> limit is perl or PDL. Since the PDL routines for growing
>>>>> a new piddle use 4byte ints for their sizes (rather than
>>>>> size_t objects), it is pretty clear that there is a bug
>>>>> in the PDL allocation stuff if perl can handle the longer
>>>>> strings.
>>>>>
>>>>> --Chris
>>>> _______________________________________________
>>>> Perldl mailing list
>>>> [email protected]
>>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
