On Fri, 2010-12-31 at 10:11 +0000, Chris Morley wrote: > So I found one problem with the g10 l2 patch : > p-1 is accepted as p0 even if I check for p < 0 > The reason is when the p number is converted to an integer: > > p_int = (int) (block->p_number + 0.0001); > > -1 +.0001 is (rightfully) truncated to zero. > If I remove the +0.0001 then -1 stays -1 and the error is > caught. > My question is the reason for the 0.0001 and can I remove > it without problems ? > Well, no. The reason for it seems to be that floating point numbers come with some inaccuracies. So the number 1 in floating point might be 1.0000001 or 0.99999998 or something like that, depending on the CPU architecture and/or the operations that were done to the number before. However, when converted to int the 1.0000001 converts as 1 and the 0.99999998 converts as 0. And thus we have to add a "safe" value to it before casting it. 0.0001 is pretty safe for doubles and also seems to be safe for floats. This is also the reason why floating point numbers cannot be compared with ==. They often wouldn't match, even if it's actually the same number, due to inaccuracies.
-- Greetings Michael. ------------------------------------------------------------------------------ Learn how Oracle Real Application Clusters (RAC) One Node allows customers to consolidate database storage, standardize their database environment, and, should the need arise, upgrade to a full multi-node Oracle RAC database without downtime or disruption http://p.sf.net/sfu/oracle-sfdevnl _______________________________________________ Emc-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/emc-developers
