> Every standby-mhz value Arduino is sending has been previously formatted so > that it has at most three decimals, something like that: 139.775 or 129.675. > Problem is FGFS shows /instrumentation/comm/frequencies/standby-mhz as > 139.7749939 or 129.6750031. There's something wrong with that and I have no > idea what it is. It looks like FGFS is receiving 139.775 and stores it as > 139.7749939 (some float rounding up) ... but why? Suggestions? It's an issue with the finite precision of floating point variables. Everyone is suprised when first seeing this. Only values which happen to be a sum of "binary fractions" (e.g. 1/2 + 1/4 + 1/8) can be represented accurately. Everything else, even simple _decimal_ values such as "0.1" or "0.775" cannot be represented exactly in _binary_. Usually this doesn't matter, since when you print a value to screen - or to a string buffer, you'll limit the precision. Something like 0.7749998175 will be printed as "0.775" if you limit to 3 digits. Obviously the property browser doesn't limit to 3 digits, but shows more, hence you see a difference.
See here: http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems Also, the properties use double floating points (64bit). You're protocol uses float (32bit) - so you loose some precision, which makes the effect slightly worse. cheers, Thorsten ------------------------------------------------------------------------------ Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf _______________________________________________ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
