ErikBPF opened a new pull request, #7580: URL: https://github.com/apache/datafusion-comet/pull/7580
## Which issue does this PR close? Closes #7442. ## Rationale for this change At integer `truncate` widths above 2^30, Iceberg Java's `(v % W) + W` overflows a 32-bit `int`, while the format spec and iceberg-rust compute `v - (v % W)`. Every Iceberg release through 1.12.0 overflows, and a release carrying apache/iceberg#18112 returns the spec value, so the expected answer depends on the Iceberg jar the user runs. Comet cannot choose from the Spark profile, because users pick their Iceberg runtime independently. ## What changes are included in this PR? - `CometIcebergTruncate` declines an integer `truncate[W]` with `W > 2^30` for `tinyint`, `smallint`, `int` and `date`, so Spark evaluates the call with the user's jar. Long, string and binary truncation stays native: `truncateLong` uses long arithmetic and `W` is an `int`, so it cannot overflow at these widths. - A new write trigger declines a native Iceberg write whose output partition spec has an integer `truncate[W]` field above that width. The native writer would write the spec value while the user's jar writes the overflowed one into the same table. - The Rust test comment now attributes the overflow to Java (apache/iceberg#18111, fix in apache/iceberg#18112) and cites the spec, keeping `i32::MIN` / `i64::MIN` as the iceberg-rust cases. - The user guide states the limit in the system-functions section and in the write eligibility table, and `getUnsupportedReasons` lists it. ## How are these changes tested? - Support-level assertions at `2^30` and above for `int`, and for `long` at the same width. - An expression fallback test that compares Comet's answer with Spark's for `truncate(2000000000, i32)`, and keeps `Int.MaxValue` coverage for `long` and `string`. - Write-detection tests for a `truncate(2000000000, id)` partition spec (fall-back) and for the maximum safe width (Compatible). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
