Here is how I arrived at this:
var res:Number;
var a:Number = 5;
var b:Number = 4.8;
res = a - b;
trace("result: " + res + ", a: " + a + ", b: " + b);
The trace clearly shows that the value of a is 5, and the value of b
is 4.8. However the end result is clearly shown as 0.2000000000000018.
Ok, I wanted to create a workaround where I will make sure that b is
really 4.8, and I used toPecision() method.
var b1:String = b.toPrecision(2);
var b2:Number = Number(b1);
trace("b1: " + b1 + ", b2: " + b2);
res = a - b2;
trace("result: " + res + ", b1: " + b1 + ", b2: " + b2);
Again with same result. b2 was traced as 4.8, but the end result still
showing 0.2000000000000018. As this can of course break all further
calculations I had to make sure result is really holding a value that
it should, based on all mathematical logic. So I did this:
var res1:String = res.toPrecision(2);
res = Number(res1);
trace("result: " + res);
I got the expected result and the value of res is now 0.2.
This is however very ugly, in case I'm not doing something wrong.
Every time we expect a floating point value in our calculation we have
to handle it to ensure the proper value is calculated, because what
Flash calculates is just wrong. Also, every intermediate calculation
has to be stored in a variable converted to String with toPrecision
and back to Number for future use. I really don't know how to properly
handle this and would appreciate any advice. Imagine the shopping cart
system with this sort of unpredictable behavior.
For the end, here's one more weirdness:
var c:Number = 488.8;
trace("c: " + c + " , c.toPrec: " + c.toPrecision(2));
Traces out c: 4.8, c.toPrec: 4.9e+2
Thanks,
Alen
var c:Number = 488.8;
