wombatu-kun opened a new issue, #20229: URL: https://github.com/apache/hudi/issues/20229
### Bug Description **What happened:** With the column stats index enabled, a `datediff(end, col)` filter, i.e. one with the column as the start date, can prune a file that holds matching rows. `datediff(end, start)` is `end - start`, which reverses the ordering of `start`, yet data skipping treats it as order-preserving in both arguments (`BaseHoodieCatalystExpressionUtils`). A "last N days" filter like `datediff(current_date(), dt) <= 7` is thus checked against the file's oldest date, and a file that also holds recent rows is pruned. The partition stats index goes through the same translation. **What you expected:** The matching row returned. **Steps to reproduce:** ```sql CREATE TABLE t (id INT, dt DATE) USING hudi TBLPROPERTIES (primaryKey = 'id', 'hoodie.metadata.index.column.stats.enable' = 'true'); INSERT INTO t VALUES (1, DATE'2024-01-01'), (2, DATE'2024-01-09'); SELECT id FROM t WHERE datediff(DATE'2024-01-10', dt) <= 7; -- expected: 2 -- actual: no rows ``` ### Environment **Hudi version:** master (1.3.0-SNAPSHOT) **Query engine:** Spark 3.5 **Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`, `hoodie.enable.data.skipping=true` (default) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
