Kim,
Uhhh that doesnt sound right to me. I subtracted 3 different values from
another value and got the same result twice. Also two of the results are
flat out wrong.
I could understand if I had used some kind of hokey notation but its a
straight up math problem X-Y. When performed in a calculator application I
dont get that result?
On 5/17/07, Kim Gräsman <[EMAIL PROTECTED]> wrote:
Hi Richard,
[EMAIL PROTECTED] wrote:
> Had something really screwy happen today, had to do some math so I
dropped
> into my DQSD bar and typed;
>
> 2156737.51 - 16123.81
>
> and it gives me the result;
>
> 2140613.6999999997
> WOW!
That's how floating-point arithmetic works on most PCs.
I can never describe it without at least confusing accuracy and
precision, so I suggest you Google for details :-)
http://www.google.com/search?q=floating-point+arithmetic
I really have a hard time understanding the details myself, but I know
it's how it's supposed to be, even if it's disconcerting.
Cheers,
- Kim
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
To unsubscribe visit:
https://lists.sourceforge.net/lists/listinfo/dqsd-users
[email protected]
http://sourceforge.net/mailarchive/forum.php?forum_id=8601
--
Richard Ahlquist
Systems Analyst
Webmaster
http://www.patentlystupid.com
http://www.pcsites.com
http://www.unlimitedbs.com
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
To unsubscribe visit:
https://lists.sourceforge.net/lists/listinfo/dqsd-users
[email protected]
http://sourceforge.net/mailarchive/forum.php?forum_id=8601
