Winfried, to have an external reference I entered your input in R (64-bit), open source statistical SW, R confirms your math. *Calc is wrong*. Below is from R-commandline:
> (3^31)-((3^31)-1)[1] 1> (3^31)-((3^31)-2)[1] 2> (3^31)-((3^31)-3)[1] 3 > Below is whtat R states about precision: All *R* platforms are required to work with values conforming to the IEC 60559 (also known as IEEE 754) standard. This basically works with a precision of 53 bits, and represents to that precision a range of absolute values from about *2e-308* to *2e+308*. It also has special values NaN <https://stat.ethz.ch/R-manual/R-devel/library/base/html/is.finite.html> (many of them), plus and minus infinity and plus and minus zero (although *R* acts as if these are the same). There are also *denormal(ized)* (or *subnormal*) numbers with absolute values above or below the range given above but represented to less precision. 2016-09-28 13:15 GMT+02:00 Winfried Donkers <w.donk...@dci-electronics.nl>: > Hi, > > I'm working on some improvements of Calc's MOD function, but I struggle > with some unexpected results when using large integer values. > As I understand IEEE754, double precision means a mantissa of 53 bits, so > the maximum correct unsigned integer value would be 2^53. > > But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, > 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, > but 0, 0 and 3. > This test is simply in Calc with e.g. =(3^31)-((3^31)-1). > I use a 64bit machine and master-version of LO. > > What am I doing wrong here? > > Winfried > > > _______________________________________________ > LibreOffice mailing list > LibreOffice@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/libreoffice > -- Luc Castermans mailto:luc.casterm...@gmail.com
_______________________________________________ LibreOffice mailing list LibreOffice@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice