On Wed, 2011-10-05 at 15:54 -0700, Pedro wrote: > Steve Edmonds wrote: > > > > Thats interesting. In Calc I get 16.4300000001676 and in Kcalc I get > > 16.43 (to 20 decimal places) > > > > That is expected. Because calculators have a limited number of functions > they use finite precision (instead of the base 2 calculations) > > This means that for this kind of calculations you are better off using a > cheap calculator than your powerful PC with a spreadsheet :)
The issue is the internal representation of floating point numbers. But often the machine errors are much less than the noise in the data. In terms of significant figures, the correct answer is 16.43 not 16.43000-00?. The extra digits beyond the 3 are dropped for significant figures. One should actually display only the correct significant figures in the final answer. > > -- > View this message in context: > http://nabble.documentfoundation.org/LOcalc-shows-an-inaccurate-output-when-use-cells-autofill-function-tp3397081p3398248.html > Sent from the Users mailing list archive at Nabble.com. > -- Jay Lozier jsloz...@gmail.com -- For unsubscribe instructions e-mail to: users+h...@global.libreoffice.org Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://listarchives.libreoffice.org/global/users/ All messages sent to this list will be publicly archived and cannot be deleted
