Sam Ruby wrote: > I find it easier to talk about real examples than abstractions. I've > done the following quickly, so forgive me if I get some detail wrong. > > A binary floating point number has 52 bits of fraction, and by > assuming an implicit leading one, they get an additional bit. This > means that 1.1 is stored as (for brevity, I'll use hex) > > [1].1999999999999 > > A conversion of that to decimal128 would be equivalent to computing > > 4953959590107545m / 4503599627370496m > > Which would produce > > 1.099999999999999866773237044981215
That's incorrect. The correct answer is 1.100000000000000088817841970012523, which is closer mathematically to 1.1. > Repeating that for 1.2 produces > > 0x13333333333333L > 5404319552844595m / 4503599627370496m > 1.199999999999999955591079014993738 This one is correct. Waldemar _______________________________________________ Es-discuss mailing list Es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss