I’ll be pleased to write some tests for it that are basically what you have below. I’d also like to tidy up the lldf branch as it currently is by rebasing it to get rid of the merge commit.
Chris, kmx, any objections? Ed From: Chris Marshall Sent: Monday, August 31, 2015 10:58 AM To: pdl-devel@lists.sourceforge.net Subject: Re: [Pdl-devel] LongLong precision (fixed) The LongLong to Double conversion problems appear to have been resolved. I would invite any PDL developers with systems having lots of RAM to test large memory operations. We need to add some tests to check for correct operation when the PDLs and index values are >=2**31. For instance, here is a problem caused by an int where a PDL_Indx should have been: > It crashes for me as does this: > > pdl> $PDL::BIGPDL = 1; > pdl> $x = zeros(byte,1024,1024,2049); > pdl> $y = $x->copy; > Segmentation fault (core dumped) > > but this is ok: > > pdl> $PDL::BIGPDL = 1; > pdl> $x = zeros(byte,1024,1024,2047); > pdl> $y = $x->copy; > pdl> q > > so there is a problem for piddles with more > than 2**31 elements. t/bigmem.t anyone? t/bigmemmap.t anyone? --Chris On 8/26/2015 17:45, kmx wrote: Hi all, After some off-list discussion with Chris I have fixed his fix in longlong-double-fix branch. It seems to work, passes all tests and is more or less ready for merging to master. Travis-CI is all green see https://travis-ci.org/PDLPorters/pdl/builds/77348194 All changes: https://github.com/PDLPorters/pdl/compare/master...longlong-double-fix#files_bucket The main idea is a new type PDL_Anyval capable of keeping values of all types without loosing any precision. Like this: typedef enum { PDL_B, PDL_S, PDL_US, PDL_L, PDL_IND, PDL_LL, PDL_F, PDL_D } pdl_datatypes; typedef struct { pdl_datatypes type; union { PDL_Byte B; PDL_Short S; PDL_Ushort U; PDL_Long L; PDL_Indx N; PDL_LongLong Q; PDL_Float F; PDL_Double D; } value; } PDL_Anyval; There are still some questions that I want to discuss here: 1/ "broken" C API (not that much, but still) Using PDL_Anyval struct (instead of plain double) has some side effects concerning C API, namely: PDL->put_offs(..) PDL->get_offs(..) PDL->get(..) PDL->get_pdl_badvalue(..) The relevant declarations in Core structure were not changed: void (*put_offs)(pdl *pdl,PDL_Indx offs, PDL_Anyval val); PDL_Anyval (*get_offs)(pdl *pdl,PDL_Indx offs); PDL_Anyval (*get)(pdl *pdl,PDL_Indx *inds); PDL_Anyval (*get_pdl_badvalue)(pdl *it); however the old code like: if( PDL->get_pdl_badvalue(a) == 0 ) { ... } does not work anymore (it does not even compile) from obvious reason. The new style should be double bad_a; ANYVAL_TO_CTYPE(bad_a, double, PDL->get_pdl_badvalue(a)); if( bad_a == 0 ) { ... } BTW: via http://grep.cpan.me/ I have found only one Zaki's module that is affected (+of course possibly a lot of code not on CPAN) There are also four newly introduced ANYVAL_* macros in pdl.h for handling PDL_Anyval from C code. #define ANYVAL_TO_SV(outsv,inany) ... #define ANYVAL_FROM_CTYPE(outany,avtype,inval) ... #define ANYVAL_TO_CTYPE(outval,ctype,inany) ... #define ANYVAL_EQ_ANYVAL(x,y) ... Which will be a pain for those who want to write their modules/program so that it works with old PDL versions (without these macros) as well as with the new version. The only recommendation I have for this scenario is the following set of defines that has to be added to the C code for compatibility with older PDL. #if PDL_CORE_VERSION < 12 #define ANYVAL_TO_SV(outsv,inany) outsv = newSVnv((NV)(inany) #define ANYVAL_FROM_CTYPE(outany,avtype,inval) outany = (PDL_Double)(inval) #define ANYVAL_TO_CTYPE(outval,ctype,inany) outval = (ctype)(inany) #define ANYVAL_EQ_ANYVAL(x,y) (x == y) #endif 2/ typemap & undef handling I am not quite sure what is the expected behaviour when undef SV argument has to be converted to PDL_Anyval. Now I convert it into a invalid PDL_Anyval { type=-1, value.B=0 } which might not be the best idea. The old behaviour was always (even for undef) use: $var = (double)SvNV($arg) The difference can be seen for example here: my $p = sequence(4); PDL::Core::set_c($p, [1], undef); print $p The old behaviour is "undefined ... warning" + setting the value to 0 (zero). The new behaviour is to croak. Of course it would be easy to change croak() to silently setting 0 value but I want to ask: what is correct/expected/best behaviour in this case? We are talking here about a typemap so we do not have any context (e.g. type of a piddle where PDL_Anyval value will be later assigned) therefore the obvious suggestion for using badval for undef is not possible as it is not clear which badval. 3/ PDL data type auto-detection Auto-detection (we have a SV and want to decide for the best PDL datatype) is implemented at couple of places. Based on perl's SV type (int/IV or double/NV) we decide for PDL_Long (on perl with 64bit int for PDL_LongLong) or PDL_Double. Simplified code works like this: if (SvIOK(sv)) { #if IVSIZE == 8 val = (PDL_LongLong)SvIV(sv) #else val = (PDL_Long)SvIV(sv) #endif } else if (SvOK(sv)) { val = (PDL_Double)SvNV(sv) /* tries also to convert strings */ } else { // do some magick in undef case } or even simpler: if (SvIOK(sv)) { #if IVSIZE == 8 val = (PDL_LongLong)SvIV(sv) #else val = (PDL_Long)SvIV(sv) #endif } else { val = (PDL_Double)SvNV(sv) /* tries also to convert strings + undef */ } This kind of works but I am open to any comments/improvements. BTW: let's have: $p = pdl( [ 1, 2, 3 ] ); # perl integers print $p; Is it correct that type of $p is always Double? although LongLong would be more than enough? Thanks in advance for any feedback. -- kmx ------------------------------------------------------------------------------ _______________________________________________ pdl-devel mailing list pdl-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pdl-devel -------------------------------------------------------------------------------- ------------------------------------------------------------------------------ -------------------------------------------------------------------------------- _______________________________________________ pdl-devel mailing list pdl-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pdl-devel
------------------------------------------------------------------------------
_______________________________________________ pdl-devel mailing list pdl-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pdl-devel
