JFinis opened a new issue, #9713: URL: https://github.com/apache/iceberg/issues/9713
### Apache Iceberg version 1.4.3 (latest release) ### Query engine Spark ### Please describe the bug 🐞 The hour partition transform returns an `int`. For very large dates, this function will overflow. Consequently this partition transform is not order preserving, resulting in incorrect partition pruning. Consider the following Spark SQL: ``` CREATE TABLE t (x timestamp) PARTITIONED BY(hour(x)) TBLPROPERTIES(`format-version`=2); INSERT INTO iceberg.sample VALUES(timestamp '250000-01-01 00:00:00'); SELECT * FROM iceberg.sample WHERE x > timestamp '2020-01-01 00:00:00'; ``` The last query will incorrect return an empty result, as the `hour(timestamp '250000-01-01 00:00:00')` is `-2120780968`, so the partition for this row incorrectly is a very low date. Thus, the partition pruning will incorrectly prune this partition in the query, as it considers it smaller than a 2020 date. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
