Mihai Budiu created CALCITE-6395:
------------------------------------
Summary: Significant precision loss when representing REAL literals
Key: CALCITE-6395
URL: https://issues.apache.org/jira/browse/CALCITE-6395
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.36.0
Reporter: Mihai Budiu
Consider this test that could be a SqlOperatorTest:
{code:java}
f.checkScalar("CAST(CAST('36854775807.0' AS REAL) AS BIGINT)",
"36854775808", "BIGINT NOT NULL");
{code}
The produced result is actually very far:
Expected: is "36854775808"
but: was "36854779904"
This big error comes from two reasons:
- Calcite uses BigDecimal values to represent floating point values, see
[CALCITE-2067]
- When converting a Float value to a BigDecimal in RexBuilder.clean(), the
following sequence is used:
{code:java}
new BigDecimal(((Number) o).doubleValue(), MathContext.DECIMAL32)
{code}
Using a DECIMAL32 math context leads to the precision loss. Just because a
Float uses 32 bits does not mean that the decimal should also use 32 bits.
The real fix is in the PR for [CALCITE-2067], but that hasn't been reviewed for
a long time, so I will submit a fix for the clean() method..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
