On Fri, Jun 15, 2012 at 06:32:51AM +0200, Etienne scratched on the wall: > The problem is: the "real" rounding error depends here on the shell > (SQLite or JSDB) calling the SQLite library.
This shouldn't be a surprise. The core SQLite API is expecting a 64-bit binary number. That's what goes in and what comes out. I have great confidence that the bits you put in (assuming they're a legal floating point number) are going to be the bits you get back out. Otherwise, you're not putting a number in, you're putting a string representation of a number in. In your case, that's also what you're asking for back out. If you actually store the string as a sequence of characters and nothing else, again: what you put in is going to be what you get back out. But when you start to covert things between strings and binary numbers, especially complex floating point representations, it shouldn't be a big surprise that different code out there does it differently. As for output, I'm pretty sure SQLite, when asked to do a floating point to string conversion, uses the standard "%f" printf conversion. By default, it is limited to 6 decimal places. If it only sees zeros, it will roll the number back up. The number isn't "wrong", it is just being displayed in a shortened notation. -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
