A quick question here, why does 20/200.0 give me 0.10000000000000001 in cocoa I would of expected some weird rounding errors in Javascript, but I thought cocoa would be able to handle the above equation.
Like Michael said, it's due to precision (Cocoa/QC use doubles, which are 64bit floating point numbers). Javascript likely has the same precision (or possibly half as much precision? Not sure if it's float or double-backed).
If it can't help it, will it always round up, ie be 0.00000000000000001 above the correct answer. If so, i can work around it.
Rounding is a tricky subject, and you really _shouldn't_ depend on it being a constant amount off because doubles/floats use relative precision (an exponent and a magnitude) -- you'll likely just be adding more imprecision -- doubles can handle 0.00000000000000001 without any problems, but if you add that to 100000000, the result is still 100000000 because the small amount is rounded away completely. However, if you add that to 0.00000000000000001, you get 0.00000000000000002, which is twice as much as you want. 0.00000000000000020/200 would have a rounded error much much much smaller than the "standard" 0.00000000000000001 you've calculated.
(0.00000000000000020/200 gives 1.00000000000000007154e-18, in this case -- the ...7154 being the error)
-- [ christopher wright ] [email protected] http://kineme.net/
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Do not post admin requests to the list. They will be ignored. Quartzcomposer-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/quartzcomposer-dev/archive%40mail-archive.com This email sent to [email protected]
