On Jun 4, 2008, at 6:18 AM, [EMAIL PROTECTED] wrote:
>>
>> I have created a table with the following values.....
>>
>> CREATE TABLE [test] ([Id] TEXT DEFAULT(0), [PV] REAL DEFAULT(0));
>>
>> insert into [test] ("Id", "PV") values ('485314EL', '-720.237');
>> insert into [test] ("Id", "PV") values ('485314L', '-720.237');
>> insert into [test] ("Id", "PV") values ('485314LE', '720.237');
>> insert into [test] ("Id", "PV") values ('485440EL', '-720.237');
>> insert into [test] ("Id", "PV") values ('485440L', '-720.237');
>> insert into [test] ("Id", "PV") values ('485440LE', '720.237');
>> insert into [test] ("Id", "PV") values ('522116EL', '720.237');
>> insert into [test] ("Id", "PV") values ('522116L', '720.237');
>> insert into [test] ("Id", "PV") values ('522116LE', '-720.237');
>> insert into [test] ("Id", "PV") values ('522122EL', '720.237');
>> insert into [test] ("Id", "PV") values ('522122L', '720.237');
>> insert into [test] ("Id", "PV") values ('522122LE', '-720.237');
>>
>> However when I execute a SUM against this table I get an incorrect
>> value. The query I use is
>>
>> SELECT SUM([PV] ) from [test]
>>
>> The returned value is 2.2737e-13 instead of 0.
>>
>> Can anyone shed some light?
>>
Floating point round-off error. The number 720.237 does not exist as
an IEEE 754 floating point number. So an approximation has to be
used. In this case, the approximation is
720.2369999999999663486960344016551971435546875
As you add and subtract this number from itself several times, the
round-off error accumulates so that you are left with 2.2737e-13 at
the end.
D. Richard Hipp
[EMAIL PROTECTED]
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
