Hi Winfried,
On Wednesday, 2016-09-28 16:54:17 +0200, Winfried Donkers wrote:
> > > But when I try 3^31 (which is less than 2^52) and subtract 3^31-1,
> > > 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3
> > > as result, but 0, 0 and 3.
> > > This test is simply in
Hi Eike,
But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2,
3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but
0, 0 and 3.
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
That's due to approxSub() being used for operator '-',
Hi Winfried,
On Wednesday, 2016-09-28 13:15:53 +0200, Winfried Donkers wrote:
> I'm working on some improvements of Calc's MOD function, but I struggle with
> some unexpected results when using large integer values.
> As I understand IEEE754, double precision means a mantissa of 53 bits, so the
Winfried, to have an external reference I entered your input in R
(64-bit), open source
statistical SW, R confirms your math. *Calc is wrong*. Below is from
R-commandline:
> (3^31)-((3^31)-1)[1] 1> (3^31)-((3^31)-2)[1] 2> (3^31)-((3^31)-3)[1] 3
>
Below is whtat R states about precision:
Hi,
I'm working on some improvements of Calc's MOD function, but I struggle with
some unexpected results when using large integer values.
As I understand IEEE754, double precision means a mantissa of 53 bits, so the
maximum correct unsigned integer value would be 2^53.
But when I try 3^31
