voonhous commented on issue #20449: URL: https://github.com/apache/hudi/issues/20449#issuecomment-6466941036
### Plan: one issue and one fix PR per failure mechanism Each issue covers one failure mechanism and lists every operator it affects; data types are test cases inside it. A fix PR closes one issue and turns one bug family in the #20439 oracle green. Splitting by type would scatter one fix (integral overflow spans int, bigint and smallint), and splitting by operator would bundle unrelated fixes (Multiply has both a sign bug and an overflow bug). | Failure mechanism | Oracle families | Issue | Fix PR | | --- | --- | --- | --- | | Casts that wrap around or overflow into NULL | `narrowing_cast`, `decimal_cast` | #19445 | #19475 | | Negative constant reverses order in Multiply/Divide | `negative_factor` | #19445 | #20457 | | Collation-changing string casts (Spark 4) | not generated | #19445 | none | | `LIKE 'prefix%'` translated as equality on the prefix | `like_prefix` | #20430 | #19169 | | Missing column or partition stats | `missing_stats_compound`, `missing_partition_stats` | #20296 | #20328 | | String parsing and non-sortable date formats | `string_to_datetime`, `non_sortable_format`, `to_unix_timestamp` | #19481 | none | | Integral overflow without ANSI in Add, Multiply, ShiftLeft | `add_overflow`, `multiply_overflow`, `shiftleft_overflow` | to file | planned | | BitwiseOr with a constant is not monotonic | `bitwise_or` | to file | none | | Logarithms of zero or negative bounds give NULL | `log_null_bound` | to file | none | | Time-zone conversions across DST transitions | `dst_fall_back`, `to_utc_timestamp_gap` | to file | none | | `from_unixtime` for years outside 0000-9999 | `from_unixtime_years` | to file | none | #19445 spans three mechanisms, so it closes when the last of #19475, #20457 and the collation fix lands. ### How the oracle verifies it - Runs each predicate over the raw rows with Spark; any file or partition holding a TRUE row must survive column stats and partition stats pruning. - 9 column types with edge values, nulls and missing stats; every whitelisted transform in every comparison shape and in AND/OR/NOT trees; ANSI on/off x 3 time zones; fixed seed. - One test per family, one for unattributed violations (a new mechanism fails the build), one sanity test. Each failure prints its smallest counterexample. ### Gaps in the oracle itself | Gap | Ticket | | --- | --- | | Partition stats exceptions are only logged, though they fail the query | none | | No BOOLEAN/BINARY cast targets; no template that uses one column twice | none | | float, boolean, binary, nested and collated columns; NaN | #18754 / #18792 (NaN) | | Expression, record-level and bloom indexes | #20255 / #20256 (expression index) | | MOR log files, time travel, bootstrap | #20443 / #20451, #20266 / #20267 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
