voonhous opened a new pull request, #20439: URL: https://github.com/apache/hudi/pull/20439
### Describe the issue this Pull Request addresses Refs #20296. Data skipping keeps dropping files that hold matching rows, and each fix so far shipped with hand-written fixtures. This adds the oracle test proposed in that thread: the raw data is the reference answer, so every file or partition holding a row for which Spark evaluates the predicate to TRUE must survive pruning. ### Summary and Changelog - New `TestDataSkippingOracle` (test-only, one file) with a column stats test and a partition stats test. - Real parquet files; stats read from their footers and turned into metadata records the way the writer does, then fed to the production `ColumnStatsIndexSupport#computeCandidateFileNames` and `PartitionStatsIndexSupport#prunePartitions`. - Predicates cover every arm of the order-preserving whitelist in every comparison shape and in AND/OR/NOT trees, passed through the optimizer like a real query. - Dimensions: ANSI on/off, three session time zones, simulated missing stats, index version V1/V2. - Known failures on master are listed in `KNOWN_VIOLATIONS`, each pointing at its issue. An unknown failure fails the test with a minimal counterexample. A fix PR deletes its entries. <details><summary>What it finds on master</summary> Everything in the #20296 thread, plus classes not reported there yet: - `to_utc_timestamp(ts, 'America/New_York')` inverts order in the spring-forward gap - `i * k` overflow with a positive factor (ANSI off) - `log1p` / `log2` of a negative min produce NULL bounds - `from_unixtime(l, 'yyyy-MM-dd HH:mm:ss')` for years outside 0000-9999 (a format allowlist alone does not fix it) - missing stats under OR and NOT, not only AND - narrowing and decimal casts from timestamp, double and decimal `left`, `to_date` and `to_timestamp` are rewritten by Spark before Hudi sees them, so those whitelist arms never reach the matcher; the format-less forms arrive as `Cast`. Not covered: expression index, MOR log-file stats, NaN (#18792), nested/float/binary/boolean/timestamp_ntz columns. </details> ### Impact None for users. Adds about 65 s to the Spark 3.5 test job. ### Risk Level none ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
