voonhous commented on issue #19632: URL: https://github.com/apache/hudi/issues/19632#issuecomment-5572611120
Current status in #19836 at `df61fa197caa`: the reported numeric coercion fixes and the subsequent arithmetic/decimal fixes are implemented. "Matches" below means matching Spark for the tested cases and settings, not unrestricted SQL parity. | Coercion case | Current behavior | Spark parity | | --- | --- | --- | | Integral/floating comparisons, either order | Widens using the active ANSI rules | Matches tested cases on 3.5.5 and 4.1.1 | | Numeric `IN`, `<=>`, and `coalesce`, including nulls | Converts to a common numeric type | Matches tested cases on both | | Integral arithmetic and division | Widens operands; non-decimal `/` uses Double | Matches tested cases on both | | Decimal arithmetic with decimal/integral/floating/null operands | Uses Spark's decimal promotion rules without unnecessary common-type casts | Matches tested cases on both | | Arithmetic overflow and division by zero | Propagates arithmetic exceptions; retains Spark's non-ANSI behavior | Matches tested cases on both | | `DECIMAL(38,0)` vs `DECIMAL(38,18)` | Uses the active version's decimal conversion | Matches tested cases, including both Spark 4.1.1 legacy decimal settings | | High-precision decimal/integer comparisons | Applies Spark's decimal comparison rules before generic widening | Both reported mismatches fixed; matches tested cases on both | Both additional high-precision mixed comparison mismatches are fixed. The extreme decimal/decimal example in #19860 also matches the tested Spark 4.1.1 ANSI/legacy combinations. Verification: 28 Spark 3.5 utility tests passed, 360 SQL-parity assertions passed per version (3.5.5 and 4.1.1), and scalastyle is clean. The full Spark 4.1 suite remains unverified because of cached Scala-incompatible Hudi dependencies; the full procedure integration suite was not rerun at this head. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
