On 5/09/2007 10:13 AM, John Stanton wrote:
John Machin wrote:
On 5/09/2007 6:18 AM, John Stanton wrote:
These are regular floating point numbers, and behave accordingly.
Utter nonsense. round(98926650.5, 1) -> 98926650.5000001 is a BUG.
Precisely, As I said, regular floating point.
"regular floating point" does not cause such bugs; read the remainder of
my post. The bug is in sqlite or in some library routine that it is using.
98926650.5 is representable EXACTLY in "regular" floating point.
The round function should calculate a scale factor (10.0) in this
case, and then:
shifted = 98926650.5 * 10.0 // i.e. 989266505.0, with no loss of
precision
floored = floor(shifted) // result: 989266505.0 (no change)
answer = floored / 10.0 // result: 98926650.5, exactly
And here, using Python as a calculator and exemplar, is what should
happen with Selena's second case:
| >>> original = 85227887.01
| >>> original
| 85227887.010000005
| >>> shifted = original * 10.0
| >>> shifted
| 852278870.10000002
| >>> floored = float(int(shifted))
| >>> floored
| 852278870.0
| >>> import math
| >>> math.floor(shifted)
| 852278870.0
| >>> answer = floored / 10.0
| >>> answer
| 85227887.0
| >>>
Serena Lien wrote:
I have read some of the postings/faq about the difficulties with the
round
function, when certain numbers do not have a finite representation in
binary, which SQLite uses. eg 9.95 is actually 9.9499...etc so that
round(
9.95, 1) rounds down.
But, I have found several numbers which don't get rounded at all,
and in
fact return more decimal places!
round(98926650.5, 1) -> 98926650.5000001
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
