It’s surprising to me that 1004.3e0 should have type DOUBLE or REAL; I would expect it to have type DECIMAL(5, 1), where it can be represented exactly.
> On Jan 30, 2024, at 2:53 PM, Mihai Budiu <mbu...@gmail.com> wrote: > > e0 shows that these are DOUBLE values. > Moreover, power returns a DOUBLE value. > In FP the result is the wrong one, but that's the semantics of the power > function in FP. > > Mihai > ________________________________ > From: Julian Hyde <jhyde.apa...@gmail.com> > Sent: Tuesday, January 30, 2024 2:50 PM > To: dev@calcite.apache.org <dev@calcite.apache.org> > Subject: Re: Storing RexLiteral as BigDecimal values > > The inputs are decimals, and the correct answer is 1008618.49, also a > decimal, and cannot be exactly represented as a binary floating point. I’m > not sure why in this case you want a binary floating point. > >> On Jan 30, 2024, at 2:46 PM, Mihai Budiu <mbu...@gmail.com> wrote: >> >> I am evaluating this expression: SELECT power(1004.3e0, 2e0) >> The result in Java, or Postgres, when formatted as a string, is >> 1008618.4899999999 >> The result produced by the Calcite simplification code is 1008618.49 >> The simplification code can produce RexLiterals - that's where this would be >> useful. >> This rounding error is not really necessary. >> >> Mihai >> ________________________________ >> From: Julian Hyde <jhyde.apa...@gmail.com> >> Sent: Tuesday, January 30, 2024 2:40 PM >> To: dev@calcite.apache.org <dev@calcite.apache.org> >> Subject: Re: Storing RexLiteral as BigDecimal values >> >> Can you give a scenario where a RexLiteral should have a double value? >> >>> On Jan 30, 2024, at 2:36 PM, Mihai Budiu <mbu...@gmail.com> wrote: >>> >>> Hello, >>> >>> I have a question about the representation of RexLiteral values. >>> Currently DOUBLE-valued literals are represented using a BigDecimal. >>> This causes small rounding errors, introduced in the RexBuilder.clean() >>> function. >>> This causes FP expressions that are evaluated at compilation-time to >>> produce results that are slightly off from the same expressions that may be >>> evaluated at runtime, for no real reason. For example, I am running some >>> Postgres tests for FP values, and they fail because of this small >>> difference. >>> >>> I know that FP values cannot be compared for equality, and tests are >>> supposed to have some slack, but I think that this particular rounding >>> error is not necessary. >>> >>> Why can't RexLiteral actually store a Double value internally when the type >>> is Double? >>> Is this a bug or is there a deeper reason for this representation? >>> If it's a bug I can file a JIRA issue and probably fix it. >>> >>> Thank you, >>> Mihai >> >
