I am evaluating this expression: SELECT power(1004.3e0, 2e0) The result in Java, or Postgres, when formatted as a string, is 1008618.4899999999 The result produced by the Calcite simplification code is 1008618.49 The simplification code can produce RexLiterals - that's where this would be useful. This rounding error is not really necessary.
Mihai ________________________________ From: Julian Hyde <[email protected]> Sent: Tuesday, January 30, 2024 2:40 PM To: [email protected] <[email protected]> Subject: Re: Storing RexLiteral as BigDecimal values Can you give a scenario where a RexLiteral should have a double value? > On Jan 30, 2024, at 2:36 PM, Mihai Budiu <[email protected]> wrote: > > Hello, > > I have a question about the representation of RexLiteral values. > Currently DOUBLE-valued literals are represented using a BigDecimal. > This causes small rounding errors, introduced in the RexBuilder.clean() > function. > This causes FP expressions that are evaluated at compilation-time to produce > results that are slightly off from the same expressions that may be evaluated > at runtime, for no real reason. For example, I am running some Postgres tests > for FP values, and they fail because of this small difference. > > I know that FP values cannot be compared for equality, and tests are supposed > to have some slack, but I think that this particular rounding error is not > necessary. > > Why can't RexLiteral actually store a Double value internally when the type > is Double? > Is this a bug or is there a deeper reason for this representation? > If it's a bug I can file a JIRA issue and probably fix it. > > Thank you, > Mihai
