[sqlite] Setting Precision for Floating Point data
After experiencing some difficulty with comparing dates stored internally as a floating point I was informed about the following. "... due to the way floats are stored in computers, '=' isn't really a good choice of an operator for them. Instead of Date = 38953.5890509 you should use Date > 38953.5890508 AND Date < 38953.5890510" Is there any way to set a precision value? 10 or 12 even 8 decimal places would be fine in this instance for SQLite "REAL" data so that what is "seen" can be compared without surprises? ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Setting Precision for Floating Point data
Use integers if you want to assign a specific scale and precision. Because floating point numbers are an approximation you can enforce a certain precision by calculating differences. You cannot use equality with FP but you can decide that equality is when (A - B) < |N| where N is the precision and || absolute value. MoDementia wrote: > After experiencing some difficulty with comparing dates stored internally as > a floating point I was informed about the following. > > "... due to the way floats are stored in computers, '=' isn't really a good > choice of an operator for them. > Instead of > > Date = 38953.5890509 > > you should use > > Date > 38953.5890508 AND Date < 38953.5890510" > > Is there any way to set a precision value? 10 or 12 even 8 decimal places > would be fine in this instance for SQLite "REAL" data so that what is "seen" > can be compared without surprises? > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Setting Precision for Floating Point data
On Fri, May 30, 2008 at 08:16:51AM -0500, John Stanton scratched on the wall: > Use integers if you want to assign a specific scale and precision. > Because floating point numbers are an approximation you can enforce a > certain precision by calculating differences. > You cannot use equality with FP You can, but you might not get the results you expect. > but you can decide that equality is when (A - B) < |N| where N > is the precision and || absolute value. That would be: |(A-B)| < N -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "'People who live in bamboo houses should not throw pandas.' Jesus said that." - "The Ninja", www.AskANinja.com, "Special Delivery 10: Pop!Tech 2006" ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
