voonhous opened a new issue, #19632:
URL: https://github.com/apache/hudi/issues/19632
**Describe the problem**
`HoodieProcedureFilterUtils.applyTypeCoercion` handles a Long output column
compared against an Int literal by casting the **column down to Int**
(`Cast(boundRef, IntegerType)`) instead of widening the literal to Long. For
values beyond Int range this gives wrong results: under non-ANSI Spark the
value wraps (false positives/negatives), under ANSI Spark the overflow error is
swallowed by `evaluateFilter`'s per-row `Try` and the row is silently dropped.
Reachable from any procedure with a Long output column and a `filter`
argument, e.g. `show_fsview_all(filter => "data_file_size > 1000")` silently
drops every file slice over 2 GB.
**To reproduce**
Evaluate `ts > 2000` over rows with `ts = 3000000000L` via
`HoodieProcedureFilterUtils.evaluateFilter`: returns no rows on both Spark 3
(non-ANSI) and Spark 4 (ANSI). A pinned test documenting this exists in
`TestHoodieProcedureFilterUtils` ("coerces Long columns against integer
literals") since #19161.
**Suggested fix**
Widen the literal instead: `Cast(literal, LongType)`. Flip the pinned
assertion so the big-value row is kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
