Hi Jarle,
I can confirm that this works, so far, even for >4GB. Excellent news!
Cheers
Ingo
On 08/04/2010 04:43 PM, Jarle Brinchmann wrote:
> Hi Chris,
>
> It seems sufficient (to me) to merely change pdl_grow in pdlhash.c. This at
> least seems to work for me now with up to 4Gb pdls (after which my machine
> grinds to a halt). I merely changed the int's to STRLENs and put a couple of
> casts in. Seems ok but I haven't had a chance to test it extensively as my
> computer has 4Gb memory and I have other things to do :) However PDL passes
> all tests with this small modification and since the modification is all
> within one subroutine I believe it should be fine.
>
> PS: I am not checking this into the repository for now - it needs some
> independent testing.
>
> Cheers,
> Jarle.
>
>
> void pdl_grow (pdl* a, int newsize) {
>
> SV* foo;
> HV* hash;
>
> STRLEN nbytes;
> STRLEN ncurr;
> STRLEN len;
>
> if(a->state& PDL_DONTTOUCHDATA) {
> die("Trying to touch data of an untouchable (mmapped?) pdl");
> }
>
> if(a->datasv == NULL)
> a->datasv = newSVpv("",0);
>
> foo = a->datasv;
>
> nbytes = (STRLEN) newsize * pdl_howbig(a->datatype);
>
> ncurr = SvCUR( foo );
> if (ncurr == nbytes)
> return; /* Nothing to be done */
>
> /* We don't want to do this: if someone is resizing it
> * but wanting to preserve data.. */
> #ifdef FEOIJFOESIJFOJE
> if (ncurr>nbytes) /* Nuke back to zero */
> sv_setpvn(foo,"",0);
> #endif
> if(nbytes> (1024*1024*1024)) {
> SV *sv = get_sv("PDL::BIGPDL",0);
> if(sv == NULL || !(SvTRUE(sv)))
> die("Probably false alloc of over 1Gb PDL! (set $PDL::BIGPDL = 1 to
> enable)");
> fflush(stdout);
> }
>
> {
> void *p;
> p = SvGROW ( foo, nbytes );
> SvCUR_set( foo, nbytes );
> }
> a->data = (void *) SvPV( foo, len ); a->nvals = newsize;
> }
>
>
>
> On 4 Aug 2010, at 15:52, Chris Marshall wrote:
>
>> On 8/4/2010 6:22 AM, Ingo Schmid wrote:
>>> first of all, thanks for the many replies. I was not aware that this
>>> issue was unknown. I can try debugging the issue, I have access to
>>> enough memory, but little to no knowledge of perls internals.
>>>
>>>
>>> I ran the following test:
>>> for $i (0..2**27) { $str.='abcdefghijklmnopqrstuvwxyz0123456789';},
>>> that's a bit more than 2**32 (4GB). Took a few seconds to run.
>>> Then
>>> perldl> p length ($str)
>>> 4831838244
>>>
>>> perldl> p length ($str)/1024/1024/1024
>>> 4.50000003352761
>>>
>>> So I conclude it is not an underlying perl/string limitation, correct?
>>> Ingo
>>
>> Thanks for running the check. It appears confirmed that
>> this is a limitation of the current PDL allocation routines
>> that call the perl api SvGROW() but with a size as an
>> int rather than STRLEN type. That puts the limit at 2**31-1
>> for piddle sizes.
>>
>> The fix will be to change the usage of the allocation to
>> use the proper STRLEN type. Unfortunately, it is intimately
>> related to the working of PDL at the lowest level so the
>> change may break things elsewhere that use the piddles.
>>
>> It might be possible to have a shorter term fix with the
>> int type replaced by unsigned int to push the limit to
>> 4GB per piddle. Since it is the same word length, that
>> could improve things in the short term.
>>
>> However, we're in the final stages of the pre PDL-2.4.7
>> release process so this might have to wait until after
>> August to be looked at in more detail. In the meantime,
>> I'll open a feature request on sf.net for the support
>> of larger piddles.
>>
>> Cheers,
>> Chris
>>
>>> PS: My machine is unstable gentoo ~amd64, we have ubuntu boxes also.
>>>
>>> uname -a
>>>
>>> Linux spectre 2.6.33-gentoo-r2 #4 SMP Thu Jul 29 12:26:35 CEST 2010
>>> x86_64 Intel(R) Xeon(R) CPU W3520 @ 2.67GHz GenuineIntel GNU/Linux
>>>
>>> 12GB RAM
>>>
>>> perl -V:
>>>
>>>
>>> Summary of my perl5 (revision 5 version 12 subversion 1) configuration:
>>>
>>> Platform:
>>> osname=linux, osvers=2.6.33-gentoo, archname=x86_64-linux
>>> uname='linux spectre 2.6.33-gentoo #2 smp tue apr 6 10:24:11 cest
>>> 2010 x86_64 intel(r) xeon(r) cpu w3520 @ 2.67ghz genuineintel gnulinux '
>>> config_args='-des -Duseshrplib -Darchname=x86_64-linux
>>> -Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O2 -pipe -march=core2
>>> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
>>> -Dprefix=/usr -Dsiteprefix=/usr -Dvendorprefix=/usr
>>> -Dprivlib=/usr/lib64/perl5/5.12.1
>>> -Darchlib=/usr/lib64/perl5/5.12.1/x86_64-linux
>>> -Dsitelib=/usr/lib64/perl5/site_perl/5.12.1
>>> -Dsitearch=/usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
>>> -Dvendorlib=/usr/lib64/perl5/vendor_perl/5.12.1
>>> -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
>>> -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
>>> -Dsiteman1dir=/usr/share/man/man1 -Dsiteman3dir=/usr/share/man/man3
>>> -Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3
>>> -Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.12.1 -Dlocincpth=
>>> -Duselargefiles -Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost
>>> -dperladmin=r...@localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm
>>> -Di_ndbm -Di_gdbm -Di_db -Dinc_version_list=5.12.0 5.12.0/x86_64-linux
>>> -Dusrinc=/usr/include -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64'
>>> hint=recommended, useposix=true, d_sigaction=define
>>> useithreads=undef, usemultiplicity=undef
>>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>>> use64bitint=define, use64bitall=define, uselongdouble=undef
>>> usemymalloc=n, bincompat5005=undef
>>> Compiler:
>>> cc='x86_64-pc-linux-gnu-gcc', ccflags ='-fno-strict-aliasing -pipe
>>> -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
>>> optimize='-O2 -pipe -march=core2 -fomit-frame-pointer -msse4
>>> -msse4.1 -msse4.2 -mcx16 -msahf',
>>> cppflags='-fno-strict-aliasing -pipe -fstack-protector'
>>> ccversion='', gccversion='4.4.4', gccosandvers=''
>>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>>> lseeksize=8
>>> alignbytes=8, prototype=define
>>> Linker and Libraries:
>>> ld='x86_64-pc-linux-gnu-gcc', ldflags =' -fstack-protector'
>>> libpth=/usr/local/lib64 /lib64 /usr/lib64
>>> libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
>>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>>> libc=/lib/libc-2.11.2.so, so=so, useshrplib=true,
>>> libperl=libperl.so.5.12.1
>>> gnulibc_version='2.11.2'
>>> Dynamic Linking:
>>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>>> cccdlflags='-fPIC', lddlflags='-shared -O2 -pipe -march=core2
>>> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
>>> -fstack-protector'
>>>
>>>
>>> Characteristics of this binary (from libperl):
>>> Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
>>> USE_64_BIT_ALL
>>> USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO
>>> USE_PERL_ATOF
>>> Locally applied patches:
>>> 0001-gentoo_MakeMaker-RUNPATH.diff
>>> 0002-gentoo_config__over.diff
>>> 0003-gentoo_cpan__definstalldirs.diff
>>> 0004-gentoo_cpanplus__definstalldirs.diff
>>> 0005-gentoo_create-libperl-soname.diff
>>> 0006-gentoo_MakeMaker-delete__packlist.diff
>>> 0007-fixes_8d66b3f9__h2hp__fix.diff
>>> 0008-fixes_ef9df645__glob__crashes__when__File__Glob__is__empty.diff
>>>
>>> 0009-fixes_e3d01d03__Naif__calls__segfault__T__PRTOBJ__of__the__stock__typemap.diff
>>> Built under linux
>>> Compiled at Jul 26 2010 11:18:49
>>> @INC:
>>> /usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
>>> /usr/lib64/perl5/site_perl/5.12.1
>>> /usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
>>> /usr/lib64/perl5/vendor_perl/5.12.1
>>> /usr/lib64/perl5/5.12.1/x86_64-linux
>>> /usr/lib64/perl5/5.12.1
>>> /usr/lib64/perl5/site_perl
>>> /usr/lib64/perl5/vendor_perl
>>>
>>> On 08/04/2010 03:59 AM, Chris Marshall wrote:
>>>> On 8/3/2010 9:43 PM, Christian Soeller wrote:
>>>>> Is it possible to change things so that 64 bit sizes
>>>>> can be passed in the two places you identified and see
>>>>> if that works?
>>>>> I appreciate that things could still fall over in various
>>>>> PP autogenerated code pieces if ints are used for offset
>>>>> calculations in slice and other vaffine operations.
>>>> It could work. The problem is it needs someone with
>>>> a 64bit OS, lots of memory, and a willingness to
>>>> debug the issue. I don't have any *large* memory
>>>> systems at the moment. Not that I wouldn't like to
>>>> have one. :-)
>>>>
>>>> --Chris
>>>>
>>>>
>>>>> On 4/08/2010, at 12:46 PM, Chris Marshall wrote:
>>>>>
>>>>>> I took a further look at the SvGROW calls in
>>>>>> PDL/Basic/Core routines and the two that I found
>>>>>> both use int type for their sizes. That would
>>>>>> limit a piddle size to<2**31 or about 2GB.
>>>>>>
>>>>>> It looks like 64bit support for PDL may need
>>>>>> to be added to the list for the future. I don't
>>>>>> know the scope of the changes that would be
>>>>>> required to support larger PDL data objects.
>>>>>>
>>>>>> --Chris
>>>>>>
>>>>>> On 8/3/2010 8:39 PM, Chris Marshall wrote:
>>>>>>> On 8/3/2010 8:30 PM, P Kishor wrote:
>>>>>>>> On Tue, Aug 3, 2010 at 7:19 PM, Chris Marshall<[email protected]>
>>>>>>>> wrote:
>>>>>>>>> On 8/3/2010 8:01 PM, P Kishor wrote:
>>>>>>>>>> also on 64-bit Snow Leopard (Mac OS X 10.6.4)
>>>>>>>>>>
>>>>>>>>>> punk...@lucknow ~$ perl -MPDL -e '$PDL::BIGPDL=1; $x =
>>>>>>>>>> sequence(float,
>>>>>>>>>> 23171, 23171); print $x->info("%M")."\n"'
>>>>>>>>>> perl(85899) malloc: *** mmap(size=18446744071562166272) failed (error
>>>>>>>>>> code=12)
>>>>>>>>>> *** error: can't allocate region
>>>>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>>>>> Out of memory!
>>>>>>>>>> punk...@lucknow ~$
>>>>>>>>> What is perl -V?
>>>>>>> I looked at the PDL/Basic/Core stuff and it looks like
>>>>>>> if SvGROW can handle a>2GB string then, in principle,
>>>>>>> PDL should be able to handle piddles of that size.
>>>>>>>
>>>>>>> Could you see if you can create a string more than 2GB
>>>>>>> long? It might take a while but it will tell us if the
>>>>>>> limit is perl or PDL. Since the PDL routines for growing
>>>>>>> a new piddle use 4byte ints for their sizes (rather than
>>>>>>> size_t objects), it is pretty clear that there is a bug
>>>>>>> in the PDL allocation stuff if perl can handle the longer
>>>>>>> strings.
>>>>>>>
>>>>>>> --Chris
>>>>>> _______________________________________________
>>>>>> Perldl mailing list
>>>>>> [email protected]
>>>>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>> _______________________________________________
>> Perldl mailing list
>> [email protected]
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
