If the binary arithmetic/floating point is a problem, use gg in the taskbar before the equation to send it to the Google calculator which does behave like a calculator: In the taskbar: gg 2156737.51 - 16123.81 produces: http://www.google.com/search?hl=&cat=&meta=&num=&ie=utf-8&q=2156737.51+-+16123.81 which will display: 2 156 737.51 - 16 123.81 = 2 140 613.7 Fredric
________________________________ From: [EMAIL PROTECTED] on behalf of Charlie Russel Sent: Thu 5/17/2007 10:53 PM To: DQSD users mailing list Subject: Re: [DQSD-Users] Odd Math Error It's all about binary arithmetic. Spreadsheets have traditionally had this same issue. There are workarounds for it, but there's a substantial performance penalty and in most cases it's simply not enough to worry about. The number is always only off by one bit at the limit of your precision. > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:dqsd-users- > [EMAIL PROTECTED] On Behalf Of Kim Gräsman > Sent: May-17-07 10:09 PM > To: DQSD users mailing list > Subject: Re: [DQSD-Users] Odd Math Error > > Hi Richard, > > > Uhhh that doesnt sound right to me. I subtracted 3 different values > > from another value and got the same result twice. > > Oops, I didn't see your results were identical for the first two cases. > I tried it here; > > 2156737.51 - 16123.81 => 2140613.6999999997 > 2156737.51 - 16123.83 => 2140613.6799999997 <-- Note: .6_7_999... > 2156737.51 - 16123.82 => 2140613.69 > > Could you try it again to make sure you transcribed the values correctly? > > > Also two of the results are flat out wrong. > > Right, but they're pretty close to correct, which is what you get with > floating-point, if I understand things correctly. > > > I could understand if I had used some kind of hokey notation but its > > a straight up math problem X-Y. When performed in a calculator > > application I dont get that result? > > The calculator (at least the one in Windows) probably doesn't use > floating-point to store its numbers, but again, I'm a bit out of my > depth here. > > Cheers, > - Kim > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > To unsubscribe visit: > https://lists.sourceforge.net/lists/listinfo/dqsd-users > DQSD-Users@lists.sourceforge.net > http://sourceforge.net/mailarchive/forum.php?forum_id=8601 ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ To unsubscribe visit: https://lists.sourceforge.net/lists/listinfo/dqsd-users DQSD-Users@lists.sourceforge.net http://sourceforge.net/mailarchive/forum.php?forum_id=8601
<<winmail.dat>>
------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/
_______________________________________________ To unsubscribe visit: https://lists.sourceforge.net/lists/listinfo/dqsd-users DQSD-Users@lists.sourceforge.net http://sourceforge.net/mailarchive/forum.php?forum_id=8601
