The LongLong to Double conversion problems
appear to have been resolved. I would invite
any PDL developers with systems having lots
of RAM to test large memory operations.
We need to add some tests to check for correct
operation when the PDLs and index values are
>=2**31. For instance, here is a problem caused
by an int where a PDL_Indx should have been:
> It crashes for me as does this:
>
> pdl> $PDL::BIGPDL = 1;
> pdl> $x = zeros(byte,1024,1024,2049);
> pdl> $y = $x->copy;
> Segmentation fault (core dumped)
>
> but this is ok:
>
> pdl> $PDL::BIGPDL = 1;
> pdl> $x = zeros(byte,1024,1024,2047);
> pdl> $y = $x->copy;
> pdl> q
>
> so there is a problem for piddles with more
> than 2**31 elements.
t/bigmem.t anyone?
t/bigmemmap.t anyone?
--Chris
On 8/26/2015 17:45, kmx wrote:
Hi all,
After some off-list discussion with Chris I have fixed his fix in
longlong-double-fix branch. It seems to work, passes all tests and is
more or less ready for merging to master.
Travis-CI is all green see
https://travis-ci.org/PDLPorters/pdl/builds/77348194
All changes:
https://github.com/PDLPorters/pdl/compare/master...longlong-double-fix#files_bucket
The main idea is a new type PDL_Anyval capable of keeping values of
all types without loosing any precision. Like this:
typedef enum { PDL_B, PDL_S, PDL_US, PDL_L, PDL_IND, PDL_LL, PDL_F,
PDL_D } pdl_datatypes;
typedef struct {
pdl_datatypes type;
union {
PDL_Byte B;
PDL_Short S;
PDL_Ushort U;
PDL_Long L;
PDL_Indx N;
PDL_LongLong Q;
PDL_Float F;
PDL_Double D;
} value;
} PDL_Anyval;
There are still some questions that I want to discuss here:
_*1/ "broken" C API (not that much, but still)*_
Using PDL_Anyval struct (instead of plain double) has some side
effects concerning C API, namely:
PDL->put_offs(..)
PDL->get_offs(..)
PDL->get(..)
PDL->get_pdl_badvalue(..)
The relevant declarations in Core structure were not changed:
void (*put_offs)(pdl *pdl,PDL_Indx offs, PDL_Anyval val);
PDL_Anyval (*get_offs)(pdl *pdl,PDL_Indx offs);
PDL_Anyval (*get)(pdl *pdl,PDL_Indx *inds);
PDL_Anyval (*get_pdl_badvalue)(pdl *it);
however the old code like:
if( PDL->get_pdl_badvalue(a) == 0 ) { ... }
does not work anymore (it does not even compile) from obvious reason.
The new style should be
double bad_a;
ANYVAL_TO_CTYPE(bad_a, double, PDL->get_pdl_badvalue(a));
if( bad_a == 0 ) { ... }
BTW: via http://grep.cpan.me/I have found only one Zaki's module that
is affected (+of course possibly a lot of code not on CPAN)
There are also four newly introduced ANYVAL_* macros in pdl.h for
handling PDL_Anyval from C code.
#define ANYVAL_TO_SV(outsv,inany) ...
#define ANYVAL_FROM_CTYPE(outany,avtype,inval) ...
#define ANYVAL_TO_CTYPE(outval,ctype,inany) ...
#define ANYVAL_EQ_ANYVAL(x,y) ...
Which will be a pain for those who want to write their modules/program
so that it works with old PDL versions (without these macros) as well
as with the new version. The only recommendation I have for this
scenario is the following set of defines that has to be added to the C
code for compatibility with older PDL.
#if PDL_CORE_VERSION < 12
#define ANYVAL_TO_SV(outsv,inany) outsv = newSVnv((NV)(inany)
#define ANYVAL_FROM_CTYPE(outany,avtype,inval) outany =
(PDL_Double)(inval)
#define ANYVAL_TO_CTYPE(outval,ctype,inany) outval = (ctype)(inany)
#define ANYVAL_EQ_ANYVAL(x,y) (x == y)
#endif
_*2/ typemap & undef handling*_
I am not quite sure what is the expected behaviour when undef SV
argument has to be converted to PDL_Anyval. Now I convert it into a
invalid PDL_Anyval { type=-1, value.B=0 } which might not be the best
idea.
The old behaviour was always (even for undef) use:
$var = (double)SvNV($arg)
The difference can be seen for example here:
my $p = sequence(4);
PDL::Core::set_c($p, [1], undef);
print $p
The old behaviour is "undefined ... warning" + setting the value to 0
(zero). The new behaviour is to croak.
Of course it would be easy to change croak() to silently setting 0
value but I want to ask: what is correct/expected/best behaviour in
this case?
We are talking here about a typemap so we do not have any context
(e.g. type of a piddle where PDL_Anyval value will be later assigned)
therefore the obvious suggestion for using badval for undef is not
possible as it is not clear which badval.
_*3/ PDL data type auto-detection*_
Auto-detection (we have a SV and want to decide for the best PDL
datatype) is implemented at couple of places. Based on perl's SV type
(int/IV or double/NV) we decide for PDL_Long (on perl with 64bit int
for PDL_LongLong) or PDL_Double.
Simplified code works like this:
if (SvIOK(sv)) {
#if IVSIZE == 8
val = (PDL_LongLong)SvIV(sv)
#else
val = (PDL_Long)SvIV(sv)
#endif
} else if (SvOK(sv)) {
val = (PDL_Double)SvNV(sv) /* tries also to convert strings */
} else {
// do some magick in undef case
}
or even simpler:
if (SvIOK(sv)) {
#if IVSIZE == 8
val = (PDL_LongLong)SvIV(sv)
#else
val = (PDL_Long)SvIV(sv)
#endif
} else {
val = (PDL_Double)SvNV(sv) /* tries also to convert strings +
undef */
}
This kind of works but I am open to any comments/improvements.
BTW: let's have:
$p = pdl( [ 1, 2, 3] ); # perl integers
print $p;
Is it correct that type of $p is always Double? although LongLong
would be more than enough?
Thanks in advance for any feedback.
--
kmx
------------------------------------------------------------------------------
_______________________________________________
pdl-devel mailing list
pdl-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pdl-devel
------------------------------------------------------------------------------
_______________________________________________
pdl-devel mailing list
pdl-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pdl-devel
