Re: [Flashcoders] Problems when converting to Number
You're getting to the limits of the precision of a Number.
In ActionScript 3, the Number type is stored in binary, in 64-bit
double-precision IEEE floating point format:
http://en.wikipedia.org/wiki/Binary64
It uses 52 bits for the mantissa (fractional part) which gives it an effective
precision of 53 bits, equal to about 16 decimal digits of precision. The error
you're getting is in the 17th digit - no surprise at all.
-Gerry
On 2010-12-19 , at 09:55 , Anthony Pace wrote:
trace(Number('1992.2'));
//why does it output 1992.3
//I am assuming I am missing something pretty obvious
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
Oh well, after reading a couple of articles, I think I have confirmed my
suspicion; for, as I thought, it just comes down to there being too many
digits, and the floating point implementation doing some hidden
weirdness. It really sucks that you can really only expect about 15
digits of true accuracy from the Number Data type.
If I am wrong let me know.
On 12/18/2010 8:55 PM, Anthony Pace wrote:
trace(Number('1992.2'));
//why does it output 1992.3
//I am assuming I am missing something pretty obvious
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
Thanks for the response. Right as you were posting, I was posting my
own message of how I read up on the spec, and saw the same thing.
From what I have read, as you can't actually have the int portion be 9
999 999 999 999 999, because its limit is 9,007,199,254,740,992 , you
can really only trust the use of Number up to 15 digits of accuracy.
Let me know if I am wrong.
On 12/19/2010 4:33 AM, Gerry Beauregard wrote:
You're getting to the limits of the precision of a Number.
In ActionScript 3, the Number type is stored in binary, in 64-bit
double-precision IEEE floating point format:
http://en.wikipedia.org/wiki/Binary64
It uses 52 bits for the mantissa (fractional part) which gives it an effective
precision of 53 bits, equal to about 16 decimal digits of precision. The error
you're getting is in the 17th digit - no surprise at all.
-Gerry
On 2010-12-19 , at 09:55 , Anthony Pace wrote:
trace(Number('1992.2'));
//why does it output 1992.3
//I am assuming I am missing something pretty obvious
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
Anthony Pace skriver:
trace(Number('1992.2'));
//why does it output 1992.3
//I am assuming I am missing something pretty obvious
First it is parsed to binary floating point, losing accuracy and then it
is output as decimal form again, losing accuracy again.
___
Flashcoders mailing list
Flashcoders@chattyfig.figleaf.com
http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
