Re: [Flashcoders] Problems when converting to Number
Anthony Pace skriver: trace(Number('1992.2')); //why does it output 1992.3 //I am assuming I am missing something pretty obvious First it is parsed to binary floating point, losing accuracy and then it is output as decimal form again, losing accuracy again. ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
Thanks for the response. Right as you were posting, I was posting my own message of how I read up on the spec, and saw the same thing. From what I have read, as you can't actually have the int portion be 9 999 999 999 999 999, because its limit is 9,007,199,254,740,992 , you can really only trust the use of Number up to 15 digits of accuracy. Let me know if I am wrong. On 12/19/2010 4:33 AM, Gerry Beauregard wrote: You're getting to the limits of the precision of a Number. In ActionScript 3, the Number type is stored in binary, in 64-bit double-precision IEEE floating point format: http://en.wikipedia.org/wiki/Binary64 It uses 52 bits for the mantissa (fractional part) which gives it an effective precision of 53 bits, equal to about 16 decimal digits of precision. The error you're getting is in the 17th digit - no surprise at all. -Gerry On 2010-12-19 , at 09:55 , Anthony Pace wrote: trace(Number('1992.2')); //why does it output 1992.3 //I am assuming I am missing something pretty obvious ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
Oh well, after reading a couple of articles, I think I have confirmed my suspicion; for, as I thought, it just comes down to there being too many digits, and the floating point implementation doing some hidden weirdness. It really sucks that you can really only expect about 15 digits of true accuracy from the Number Data type. If I am wrong let me know. On 12/18/2010 8:55 PM, Anthony Pace wrote: trace(Number('1992.2')); //why does it output 1992.3 //I am assuming I am missing something pretty obvious ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders
Re: [Flashcoders] Problems when converting to Number
You're getting to the limits of the precision of a Number. In ActionScript 3, the Number type is stored in binary, in 64-bit double-precision IEEE floating point format: http://en.wikipedia.org/wiki/Binary64 It uses 52 bits for the mantissa (fractional part) which gives it an effective precision of 53 bits, equal to about 16 decimal digits of precision. The error you're getting is in the 17th digit - no surprise at all. -Gerry On 2010-12-19 , at 09:55 , Anthony Pace wrote: > trace(Number('1992.2')); > > //why does it output 1992.3 > //I am assuming I am missing something pretty obvious > ___ > Flashcoders mailing list > Flashcoders@chattyfig.figleaf.com > http://chattyfig.figleaf.com/mailman/listinfo/flashcoders ___ Flashcoders mailing list Flashcoders@chattyfig.figleaf.com http://chattyfig.figleaf.com/mailman/listinfo/flashcoders