What did I mean? I'm not sure. For some reason I thought that floats loose precision at near the end of their range, but even if that is true, that wouldn't explain the lose of precision that large. In any event, I retried the following code:
var t1:Number = Number.MAX_VALUE; var t3:Number = Number.MAX_VALUE - 1290; var t4:Number = Number.MAX_VALUE - 1284; var t5:Number = Math.abs(Number.MAX_VALUE - 1290); var t6:Number = Math.abs(Number.MAX_VALUE - 1284); var a1:Number = t3 - t4; var a2:Number = Math.abs(t3 - t4); var a3:Number = t5 - t6; var a4:Number = (Number.MAX_VALUE - 1290) - (Number.MAX_VALUE - 1284); var a5:Number = Math.abs(Number.MAX_VALUE - 1290) - Math.abs(Number.MAX_VALUE - 1284); var a6:Number = Math.abs((Number.MAX_VALUE - 1290) - (Number.MAX_VALUE - 1284)); The debugger reported all of the first group of variable values as "9223372036854775807 [0x7fffffffffffffff]" and the traces of these values reported all of the values as "1.79769313486231e+308" This is not the same results that I remember getting previously where t3 != t4. The second group all have values of 0. When i have time I'll look into this more. - Daniel Freiman On Wed, Jun 11, 2008 at 2:04 AM, Gordon Smith <[EMAIL PROTECTED]> wrote: > > Number.MAX_VALUE doesn't have enough precision to handle what I was > trying to do. > > What do you mean by this? A Number, being 64 bits, actually has both more > precision and more range than int, which is 32 bits. For example, in > addition to storing fractional values, it can store integers much larger > than int.MAX_VALUE. > > > > Gordon Smith > > Adobe Flex SDK Team > > > ------------------------------ > > *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On > Behalf Of *Daniel Freiman > *Sent:* Friday, June 06, 2008 2:10 PM > *To:* flexcoders@yahoogroups.com > *Subject:* Re: [flexcoders] Math.abs() Limitation? > > > > I figured it out. The trace made it obvious (as opposed the debugger which > is what I was previously using). I was using Number.MAX_VALUE, not > int.MAX_VALUE. Number.MAX_VALUE doesn't have enough precision to handle > what I was trying to do. > > Thanks, > > - Daniel Freiman > > On Fri, Jun 6, 2008 at 2:29 PM, Gordon Smith <[EMAIL PROTECTED]> wrote: > > What do you get when you trace out the following values? > > > > int.MAX_VALUE - 1290 > > Math.abs(int.MAX_VALUE - 1290) > > int.MAX_VALUE - 1284 > > Math.abs(int.MAX_VALUE - 1284) > > > > Gordon Smith > > Adobe Flex SDK Team > > > ------------------------------ > > *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On > Behalf Of *Alex Harui > *Sent:* Friday, June 06, 2008 11:18 AM > *To:* flexcoders@yahoogroups.com > *Subject:* RE: [flexcoders] Math.abs() Limitation? > > > > What if you use temporary variables? > > > ------------------------------ > > *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On > Behalf Of *Daniel Freiman > *Sent:* Friday, June 06, 2008 9:09 AM > *To:* flexcoders > *Subject:* [flexcoders] Math.abs() Limitation? > > > > According to my code the following statement returns true. > > Math.abs(int.MAX_VALUE - 1290) == Math.abs(int.MAX_VALUE - 1284) > > I tried converting everything to type Number that didn't help. Getting > ride of the abs() makes the calculation work correctly but then I don't have > the absolute value. > > Thoughts? > > - Daniel Freiman > > > > >