[
https://issues.apache.org/jira/browse/CALCITE-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-6395:
------------------------------------
Labels: pull-request-available (was: )
> Significant precision loss when representing REAL literals
> ----------------------------------------------------------
>
> Key: CALCITE-6395
> URL: https://issues.apache.org/jira/browse/CALCITE-6395
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.36.0
> Reporter: Mihai Budiu
> Priority: Minor
> Labels: pull-request-available
>
> Consider this test that could be a SqlOperatorTest:
> {code:java}
> f.checkScalar("CAST(CAST('36854775807.0' AS REAL) AS BIGINT)",
> "36854775808", "BIGINT NOT NULL");
> {code}
> The produced result is actually very far:
> Expected: is "36854775808"
> but: was "36854779904"
> This big error comes from two reasons:
> - Calcite uses BigDecimal values to represent floating point values, see
> [CALCITE-2067]
> - When converting a Float value to a BigDecimal in RexBuilder.clean(), the
> following sequence is used:
> {code:java}
> new BigDecimal(((Number) o).doubleValue(), MathContext.DECIMAL32)
> {code}
> Using a DECIMAL32 math context leads to the precision loss. Just because a
> Float uses 32 bits does not mean that the decimal should also use 32 bits.
> The real fix is in the PR for [CALCITE-2067], but that hasn't been reviewed
> for a long time, so I will submit a fix for the clean() method..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
