mbutrovich commented on issue #12048: URL: https://github.com/apache/arrow-rs/issues/12048#issuecomment-6609413607
> These results are neither the exact binary value of the input (`295161.0` is exactly 295161) nor its decimal representation. Should we decide here which of those two the cast should return? The exact binary value is how #10509 defined the reverse direction, as the float nearest the decimal's exact value. Rounding it at the target scale fixes `295161.0` and every other integer. It needs no formatting: with `x = m * 2^e`, the result is `m * 10^scale * 2^e` rounded half away from zero. It also covers `Float32` input, because widening to `f64` is exact. The shortest decimal string matches Spark instead. The two differ at `0.1` and scale 20, which gives `0.10000000000000000555` from the exact value and `0.10000000000000000000` from the string. > The share of wrong results, measured against the input's shortest decimal representation: This table counts a result as wrong when it differs from the shortest decimal string, so it assumes the answer to the question above. > One downstream effect: DataFusion compares `DECIMAL` with `DOUBLE` by casting the double to `Decimal128(30, 15)` (apache/datafusion#14272). As I read it, the cause there is DataFusion's coercion. apache/datafusion#14272 asks DataFusion to use the float type as the common type for these comparisons. DataFusion's rule cites Spark ([binary.rs](https://github.com/apache/datafusion/blob/32bd81fb226ccf1ae5df67ff340d77baa3e73668/datafusion/expr-common/src/type_coercion/binary.rs#L1381-L1395)), but Spark casts the decimal to double when comparing it with a double ([DecimalPrecisionTypeCoercion.scala](https://github.com/apache/spark/blob/cecbd5d846a2cdd891a4d9f0bda302b58b02092a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionTypeCoercion.scala#L186-L189)). > This is also what PostgreSQL and Spark (`BigDecimal(Double.toString)`) use. PostgreSQL rounds to 15 significant digits instead. `float8_numeric` formats with `%.*g` and `DBL_DIG` ([numeric.c](https://github.com/postgres/postgres/blob/884aca9b96989b9bc1b65d42e731e90f098ee2ee/src/backend/utils/adt/numeric.c#L4570)). For Spark, Comet already implements this `ryu` approach in its own crate ([numeric.rs](https://github.com/apache/datafusion-comet/blob/79b5ddd50ba2e3f605399d641e4c0b3be84c20ef/native/spark-expr/src/conversion_funcs/numeric.rs#L1114-L1123)), so as I read it Spark compatibility doesn't depend on arrow's cast. > Below 2^53 the current results can stay unchanged. Should they? The `f64` product isn't correctly rounded below 2^53 either. `26.657349498176934` at scale 14 gives `2665734949817694`, because the product rounds to `2665734949817693.5`. The exact value and the shortest decimal string both give `2665734949817693`. In a sample of random doubles between 2^-30 and 2^30, at scales 0 to 15 and with products below 2^53, 4,785 of 872,737 were off by one this way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
