andygrove opened a new issue, #6145: URL: https://github.com/apache/datafusion-comet/issues/6145
### Describe the bug Partition values for dates and timestamps outside the range chrono and arrow can represent (beyond roughly year 262143) are handled inconsistently by the native Iceberg writer: - `day` and `hour` transforms, and `identity` on a `date` column, fall through `iceberg_partition_path.rs:199` into iceberg-rust's human-string rendering, which panics on the overflow. The panic is caught at the JNI boundary, so the task fails and the abort guard cleans up, but the write fails where iceberg-java succeeds. - `year` and `month` transforms go through arrow's `date_part`, which returns NULL for values it cannot represent. Those rows get a NULL partition value, where iceberg-java computes the real one. That is silent wrong metadata. Only reachable with extreme values such as `date_from_unix_date(100000000)` or `timestamp_micros(9000000000000000000)`, so low priority. Found by reading the code; not reproduced. ### Steps to reproduce Write rows whose date or timestamp partition source value lies beyond year 262143 into tables partitioned by `days(ts)`, `hours(ts)`, `years(ts)` and `months(ts)` with `spark.comet.iceberg.write.enabled=true`. ### Expected behavior Either the same partition values as iceberg-java, or a clean fallback. At minimum the `year`/`month` case should not write a NULL partition value. ### Additional context Found in an audit of the native Iceberg write path. Part of #5649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
