You can sort of fix this issue by rounding, but it just hides the issue and
should not be used for financial applications:
var setPrecision:Function = function(number:Number, precision:int) {
precision = Math.pow(10, precision);
return Math.round(number * precision)/precision;
}
var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on
https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript
<https://stackoverflow.com/questions/632802/how-to-deal-with-number-precision-in-actionscript>
Erik
On Feb 2, 2018, at 8:39 AM, Erik J. Thomas <[email protected]> wrote:
The Number data type, when including a decimal point is a binary floating point
number and binary floating point math can result in rounding errors like yours.
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680
<https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#680>
Adobe says this about Number:
"The Number data type uses the 64-bit double-precision format as specified by
the IEEE Standard for Binary Floating-Point Arithmetic (IEEE-754) which means
rounding errors can occur... The Number data type uses 52 bits to store the
significand, with the result that numbers that require more than 52 bits to
represent precisely, such as the fraction 1/3, are only approximations. If your
application requires absolute precision with decimal numbers, you need to use
software that implements decimal floating-point arithmetic as opposed to binary
floating-point arithmetic."
Erik
On Feb 2, 2018, at 7:22 AM, Σπύρος Αγγελόπουλος <[email protected]>
wrote:
Hi Everyone,
Can anyone verify the results of calculation below ?
var n1:Number=18.9;
var n2:Number=100.0;
var n3:Number=n1*n2; //1889.9999999999998
is it bug ?
thanks
spiros
