voonhous opened a new issue, #19860: URL: https://github.com/apache/hudi/issues/19860
### Task Description Make Hudi procedure decimal filters match ordinary Spark 4.1.1 SQL under the same configuration. Follow-up to #19836, which fixes numeric filter coercion and propagates arithmetic exceptions instead of silently dropping rows. The remaining concern is decimal comparisons whose common precision would exceed 38, such as `big > frac` with `big: DECIMAL(38,0)` and `frac: DECIMAL(38,18)`. Calling a generic wider-type helper and directly evaluating the result is not enough to establish parity with Spark's complete decimal comparison analysis. Procedure filters must also pass `validateFilterExpression` before execution. **Expected behavior to verify on Spark 4.1.1** Use a row containing `big = 1000000000000000000000000000000` (10^30) and `frac = 1.5` with the types above. Two settings matter: - `spark.sql.ansi.enabled`: controls whether decimal cast overflow throws or returns null. - `spark.sql.legacy.decimal.retainFractionDigitsOnTruncate`: an internal Spark setting, default `false`, controlling which digits the common decimal type preserves when precision exceeds 38. | Retain fractional digits | ANSI enabled | Common comparison type | Expected SQL filter outcome | | --- | --- | --- | --- | | false | false | DECIMAL(38,0) | Keeps row | | false | true | DECIMAL(38,0) | Keeps row | | true | false | DECIMAL(38,18) | Overflow becomes null; excludes row | | true | true | DECIMAL(38,18) | Throws arithmetic overflow | With the default decimal setting, Spark preserves integer capacity and rounds `1.5` to `2`; the comparison remains true. The legacy setting preserves fractional digits, leaving only 20 integer digits, so 10^30 overflows. These expectations come from source inspection, not a completed Spark 4.1.1 reproduction. Earlier Spark 4.1 helper-only observations should not be described as ordinary Spark SQL behavior. Source: [Spark 4.1.1 DecimalPrecisionTypeCoercion](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionTypeCoercion.scala), particularly `decimalAndDecimal`, `widerDecimalType`, and `bounded`. **Work and acceptance criteria** - Reproduce with a DataFrame containing the explicit decimal schema and compare `df.filter("big > frac")` with Hudi procedure filter validation and evaluation on the same rows and settings. - Align decimal comparison coercion with Spark 4.1.1's rules, using the existing Spark compatibility layer if needed. - Test all four configuration combinations, asserting validation acceptance and matching rows or overflow errors. Keep these tests in `TestHoodieProcedureFilterUtils`. - Include a rounding-sensitive comparison, such as `big = frac` with values `2` and `1.5` using the same declared types. The 10^30 example alone cannot distinguish direct decimal comparison from Spark's default rounding conversion. - Preserve Spark 3.5.5 coverage and its version-appropriate behavior. The arithmetic-exception propagation already added in #19836 remains in that PR. ### Task Type Code improvement/refactoring; test enhancement. ### Related Issues - #19836 - #19632 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
