> But....the more important questoin is "Does it make a difference to your application?"
Not really. I've talked with the PMs and such about it, and this doesn't seem to be a problem. I wanted to know how SQLite3 worked in this department, and this seems to be okay behavior for my application. The PMs and I were happy to learn about how the strategy worked. This peaks my interest though: Is adding that value in the third decimal place going to ensure that rounding will work appropriately in all cases? This is something I need to sit down and work out. I printed out the beginning of the classic paper "What Every Computer Scientist Should Know About Floating-Point Arithmetic", and tried to study it a bit, but didn't get very far. I got lost when he started talking about catastrophic + benign cancellation, which I believe is possibly related to what your'e describing??? At any rate, thank you for response, Jon ______________________________________ Jon Polfer Project Engineer - High Level Software Engineering Office Phone: 262-832-0049 (Ext. 5 for Jon Polfer) Fax: E-mail: jpol...@forceamerica.com FORCE America Inc. W229 N1433 Westwood Drive, Suite 200 Waukesha, WI 53186 www.forceamerica.com The Leading Innovator in Mobile Hydraulic Solutions -----Original Message----- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Black, Michael (IS) Sent: Wednesday, January 05, 2011 7:33 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] EXTERNAL:Re: Rounding Strategy I can understand the "expectation" that 3.05 would round to 3.1. But....the more important questoin is "Does it make a difference to your application?" Here's the difference: sqlite> select 3.05+.05 < 3.0999999999999999; 1 sqlite> select 3.05+.05 < 3.099999999999999; 0 So...if you need accuracy to to that many decimal places you're in trouble. Even Las Vegas casinos wouldn't care about losing that much. A simple floating point solution is to simply add a value in the 3rd decimal place if you onliy need 2 decimal places. This will remove the imprecision of the .5 representation. sqlite> select round(3.05+.001,1); 3.1 Michael D. Black Senior Scientist Advanced Analytics Directorate Northrop Grumman Information Systems _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
