On Jun 4, 2008, at 6:18 AM, [EMAIL PROTECTED] wrote: >> >> I have created a table with the following values..... >> >> CREATE TABLE [test] ([Id] TEXT DEFAULT(0), [PV] REAL DEFAULT(0)); >> >> insert into [test] ("Id", "PV") values ('485314EL', '-720.237'); >> insert into [test] ("Id", "PV") values ('485314L', '-720.237'); >> insert into [test] ("Id", "PV") values ('485314LE', '720.237'); >> insert into [test] ("Id", "PV") values ('485440EL', '-720.237'); >> insert into [test] ("Id", "PV") values ('485440L', '-720.237'); >> insert into [test] ("Id", "PV") values ('485440LE', '720.237'); >> insert into [test] ("Id", "PV") values ('522116EL', '720.237'); >> insert into [test] ("Id", "PV") values ('522116L', '720.237'); >> insert into [test] ("Id", "PV") values ('522116LE', '-720.237'); >> insert into [test] ("Id", "PV") values ('522122EL', '720.237'); >> insert into [test] ("Id", "PV") values ('522122L', '720.237'); >> insert into [test] ("Id", "PV") values ('522122LE', '-720.237'); >> >> However when I execute a SUM against this table I get an incorrect >> value. The query I use is >> >> SELECT SUM([PV] ) from [test] >> >> The returned value is 2.2737e-13 instead of 0. >> >> Can anyone shed some light? >>
Floating point round-off error. The number 720.237 does not exist as an IEEE 754 floating point number. So an approximation has to be used. In this case, the approximation is 720.2369999999999663486960344016551971435546875 As you add and subtract this number from itself several times, the round-off error accumulates so that you are left with 2.2737e-13 at the end. D. Richard Hipp [EMAIL PROTECTED] _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users