Hi Chris,
It seems sufficient (to me) to merely change pdl_grow in pdlhash.c. This at
least seems to work for me now with up to 4Gb pdls (after which my machine
grinds to a halt). I merely changed the int's to STRLENs and put a couple of
casts in. Seems ok but I haven't had a chance to test it extensively as my
computer has 4Gb memory and I have other things to do :) However PDL passes all
tests with this small modification and since the modification is all within one
subroutine I believe it should be fine.
PS: I am not checking this into the repository for now - it needs some
independent testing.
Cheers,
Jarle.
void pdl_grow (pdl* a, int newsize) {
SV* foo;
HV* hash;
STRLEN nbytes;
STRLEN ncurr;
STRLEN len;
if(a->state & PDL_DONTTOUCHDATA) {
die("Trying to touch data of an untouchable (mmapped?) pdl");
}
if(a->datasv == NULL)
a->datasv = newSVpv("",0);
foo = a->datasv;
nbytes = (STRLEN) newsize * pdl_howbig(a->datatype);
ncurr = SvCUR( foo );
if (ncurr == nbytes)
return; /* Nothing to be done */
/* We don't want to do this: if someone is resizing it
* but wanting to preserve data.. */
#ifdef FEOIJFOESIJFOJE
if (ncurr>nbytes) /* Nuke back to zero */
sv_setpvn(foo,"",0);
#endif
if(nbytes > (1024*1024*1024)) {
SV *sv = get_sv("PDL::BIGPDL",0);
if(sv == NULL || !(SvTRUE(sv)))
die("Probably false alloc of over 1Gb PDL! (set $PDL::BIGPDL = 1 to
enable)");
fflush(stdout);
}
{
void *p;
p = SvGROW ( foo, nbytes );
SvCUR_set( foo, nbytes );
}
a->data = (void *) SvPV( foo, len ); a->nvals = newsize;
}
On 4 Aug 2010, at 15:52, Chris Marshall wrote:
> On 8/4/2010 6:22 AM, Ingo Schmid wrote:
>>
>> first of all, thanks for the many replies. I was not aware that this
>> issue was unknown. I can try debugging the issue, I have access to
>> enough memory, but little to no knowledge of perls internals.
>>
>>
>> I ran the following test:
>> for $i (0..2**27) { $str.='abcdefghijklmnopqrstuvwxyz0123456789';},
>> that's a bit more than 2**32 (4GB). Took a few seconds to run.
>> Then
>> perldl> p length ($str)
>> 4831838244
>>
>> perldl> p length ($str)/1024/1024/1024
>> 4.50000003352761
>>
>> So I conclude it is not an underlying perl/string limitation, correct?
>> Ingo
>
>
> Thanks for running the check. It appears confirmed that
> this is a limitation of the current PDL allocation routines
> that call the perl api SvGROW() but with a size as an
> int rather than STRLEN type. That puts the limit at 2**31-1
> for piddle sizes.
>
> The fix will be to change the usage of the allocation to
> use the proper STRLEN type. Unfortunately, it is intimately
> related to the working of PDL at the lowest level so the
> change may break things elsewhere that use the piddles.
>
> It might be possible to have a shorter term fix with the
> int type replaced by unsigned int to push the limit to
> 4GB per piddle. Since it is the same word length, that
> could improve things in the short term.
>
> However, we're in the final stages of the pre PDL-2.4.7
> release process so this might have to wait until after
> August to be looked at in more detail. In the meantime,
> I'll open a feature request on sf.net for the support
> of larger piddles.
>
> Cheers,
> Chris
>
>> PS: My machine is unstable gentoo ~amd64, we have ubuntu boxes also.
>>
>> uname -a
>>
>> Linux spectre 2.6.33-gentoo-r2 #4 SMP Thu Jul 29 12:26:35 CEST 2010
>> x86_64 Intel(R) Xeon(R) CPU W3520 @ 2.67GHz GenuineIntel GNU/Linux
>>
>> 12GB RAM
>>
>> perl -V:
>>
>>
>> Summary of my perl5 (revision 5 version 12 subversion 1) configuration:
>>
>> Platform:
>> osname=linux, osvers=2.6.33-gentoo, archname=x86_64-linux
>> uname='linux spectre 2.6.33-gentoo #2 smp tue apr 6 10:24:11 cest
>> 2010 x86_64 intel(r) xeon(r) cpu w3520 @ 2.67ghz genuineintel gnulinux '
>> config_args='-des -Duseshrplib -Darchname=x86_64-linux
>> -Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O2 -pipe -march=core2
>> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
>> -Dprefix=/usr -Dsiteprefix=/usr -Dvendorprefix=/usr
>> -Dprivlib=/usr/lib64/perl5/5.12.1
>> -Darchlib=/usr/lib64/perl5/5.12.1/x86_64-linux
>> -Dsitelib=/usr/lib64/perl5/site_perl/5.12.1
>> -Dsitearch=/usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
>> -Dvendorlib=/usr/lib64/perl5/vendor_perl/5.12.1
>> -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
>> -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
>> -Dsiteman1dir=/usr/share/man/man1 -Dsiteman3dir=/usr/share/man/man3
>> -Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3
>> -Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.12.1 -Dlocincpth=
>> -Duselargefiles -Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost
>> -dperladmin=r...@localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm
>> -Di_ndbm -Di_gdbm -Di_db -Dinc_version_list=5.12.0 5.12.0/x86_64-linux
>> -Dusrinc=/usr/include -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64'
>> hint=recommended, useposix=true, d_sigaction=define
>> useithreads=undef, usemultiplicity=undef
>> useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
>> use64bitint=define, use64bitall=define, uselongdouble=undef
>> usemymalloc=n, bincompat5005=undef
>> Compiler:
>> cc='x86_64-pc-linux-gnu-gcc', ccflags ='-fno-strict-aliasing -pipe
>> -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
>> optimize='-O2 -pipe -march=core2 -fomit-frame-pointer -msse4
>> -msse4.1 -msse4.2 -mcx16 -msahf',
>> cppflags='-fno-strict-aliasing -pipe -fstack-protector'
>> ccversion='', gccversion='4.4.4', gccosandvers=''
>> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
>> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
>> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
>> lseeksize=8
>> alignbytes=8, prototype=define
>> Linker and Libraries:
>> ld='x86_64-pc-linux-gnu-gcc', ldflags =' -fstack-protector'
>> libpth=/usr/local/lib64 /lib64 /usr/lib64
>> libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
>> perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
>> libc=/lib/libc-2.11.2.so, so=so, useshrplib=true,
>> libperl=libperl.so.5.12.1
>> gnulibc_version='2.11.2'
>> Dynamic Linking:
>> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>> cccdlflags='-fPIC', lddlflags='-shared -O2 -pipe -march=core2
>> -fomit-frame-pointer -msse4 -msse4.1 -msse4.2 -mcx16 -msahf
>> -fstack-protector'
>>
>>
>> Characteristics of this binary (from libperl):
>> Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
>> USE_64_BIT_ALL
>> USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO
>> USE_PERL_ATOF
>> Locally applied patches:
>> 0001-gentoo_MakeMaker-RUNPATH.diff
>> 0002-gentoo_config__over.diff
>> 0003-gentoo_cpan__definstalldirs.diff
>> 0004-gentoo_cpanplus__definstalldirs.diff
>> 0005-gentoo_create-libperl-soname.diff
>> 0006-gentoo_MakeMaker-delete__packlist.diff
>> 0007-fixes_8d66b3f9__h2hp__fix.diff
>> 0008-fixes_ef9df645__glob__crashes__when__File__Glob__is__empty.diff
>>
>> 0009-fixes_e3d01d03__Naif__calls__segfault__T__PRTOBJ__of__the__stock__typemap.diff
>> Built under linux
>> Compiled at Jul 26 2010 11:18:49
>> @INC:
>> /usr/lib64/perl5/site_perl/5.12.1/x86_64-linux
>> /usr/lib64/perl5/site_perl/5.12.1
>> /usr/lib64/perl5/vendor_perl/5.12.1/x86_64-linux
>> /usr/lib64/perl5/vendor_perl/5.12.1
>> /usr/lib64/perl5/5.12.1/x86_64-linux
>> /usr/lib64/perl5/5.12.1
>> /usr/lib64/perl5/site_perl
>> /usr/lib64/perl5/vendor_perl
>>
>> On 08/04/2010 03:59 AM, Chris Marshall wrote:
>>> On 8/3/2010 9:43 PM, Christian Soeller wrote:
>>>> Is it possible to change things so that 64 bit sizes
>>>> can be passed in the two places you identified and see
>>>> if that works?
>>>> I appreciate that things could still fall over in various
>>>> PP autogenerated code pieces if ints are used for offset
>>>> calculations in slice and other vaffine operations.
>>>
>>> It could work. The problem is it needs someone with
>>> a 64bit OS, lots of memory, and a willingness to
>>> debug the issue. I don't have any *large* memory
>>> systems at the moment. Not that I wouldn't like to
>>> have one. :-)
>>>
>>> --Chris
>>>
>>>
>>>> On 4/08/2010, at 12:46 PM, Chris Marshall wrote:
>>>>
>>>>> I took a further look at the SvGROW calls in
>>>>> PDL/Basic/Core routines and the two that I found
>>>>> both use int type for their sizes. That would
>>>>> limit a piddle size to<2**31 or about 2GB.
>>>>>
>>>>> It looks like 64bit support for PDL may need
>>>>> to be added to the list for the future. I don't
>>>>> know the scope of the changes that would be
>>>>> required to support larger PDL data objects.
>>>>>
>>>>> --Chris
>>>>>
>>>>> On 8/3/2010 8:39 PM, Chris Marshall wrote:
>>>>>> On 8/3/2010 8:30 PM, P Kishor wrote:
>>>>>>> On Tue, Aug 3, 2010 at 7:19 PM, Chris Marshall<[email protected]>
>>>>>>> wrote:
>>>>>>>> On 8/3/2010 8:01 PM, P Kishor wrote:
>>>>>>>>> also on 64-bit Snow Leopard (Mac OS X 10.6.4)
>>>>>>>>>
>>>>>>>>> punk...@lucknow ~$ perl -MPDL -e '$PDL::BIGPDL=1; $x = sequence(float,
>>>>>>>>> 23171, 23171); print $x->info("%M")."\n"'
>>>>>>>>> perl(85899) malloc: *** mmap(size=18446744071562166272) failed (error
>>>>>>>>> code=12)
>>>>>>>>> *** error: can't allocate region
>>>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>>>> Out of memory!
>>>>>>>>> punk...@lucknow ~$
>>>>>>>> What is perl -V?
>>>>>> I looked at the PDL/Basic/Core stuff and it looks like
>>>>>> if SvGROW can handle a>2GB string then, in principle,
>>>>>> PDL should be able to handle piddles of that size.
>>>>>>
>>>>>> Could you see if you can create a string more than 2GB
>>>>>> long? It might take a while but it will tell us if the
>>>>>> limit is perl or PDL. Since the PDL routines for growing
>>>>>> a new piddle use 4byte ints for their sizes (rather than
>>>>>> size_t objects), it is pretty clear that there is a bug
>>>>>> in the PDL allocation stuff if perl can handle the longer
>>>>>> strings.
>>>>>>
>>>>>> --Chris
>>>>> _______________________________________________
>>>>> Perldl mailing list
>>>>> [email protected]
>>>>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
> _______________________________________________
> Perldl mailing list
> [email protected]
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
