andygrove opened a new pull request, #5651: URL: https://github.com/apache/datafusion-comet/pull/5651
## Which issue does this PR close? Closes #5636. Closes #5641. ## Rationale for this change The native Iceberg writer pins iceberg-rust at `3d84c81` (2026-07-29), one day before apache/iceberg-rust#2875 merged. That fix makes `PartitionSpec::partition_to_path` form-urlencode partition field names and values, which is what iceberg-java's `PartitionSpec.partitionToPath` has always done through `URLEncoder.encode`. Without it the native writer commits raw partition directories: a value containing `#` or `?` lands in the manifest as a location that iceberg-java's `S3FileIO` parses as a URI fragment or query, so a table written by Comet resolves to a different object key when read back through the standard Java path. This is the first phase-1 item of the native Iceberg writes epic (#5649). ## What changes are included in this PR? - `native/Cargo.toml` pins `iceberg` and `iceberg-storage-opendal` to `8adaa872f31549dd5ad8255848715758228038bc`, the merge commit of apache/iceberg-rust#2875. That revision has the same arrow (58.4), parquet (58.4), DataFusion (54.1), and opendal (0.57) versions as the current pin, so the lockfile change is confined to the two iceberg crates and no Comet code needed to change. The v0.10.1 release was considered and rejected: it sits on a release branch that does not contain the fix and pins DataFusion 53. - `iceberg-writes.md` drops the "partition paths are not URL-escaped" operational caveat and the eligibility table's cross-reference to it; the layouts now match. Both encoders use the application/x-www-form-urlencoded table: alphanumerics and `*-._` unescaped, space as `+`, everything else percent-encoded as UTF-8, and `null` for a null value. The commits between this revision and iceberg-rust `main` (opendal 0.58, `ObjectStorageLocationGenerator`, a GCS host fix, additional parquet writer settings) are left for the pin policy in #5645 so this change stays a targeted correctness fix. ## How are these changes tested? A new test in `CometIcebergWriteActionSuite`, "partition paths are URL-escaped like iceberg-java", writes the same rows through the native writer and the JVM writer into two tables partitioned by a string column whose values contain `/`, `#`, `?`, space, `=`, `%`, `+`, the unescaped set `*-._`, multi-byte characters, and the empty string. It asserts that the set of partition directories under each table's data location is identical, that the expected encodings (`region=a%2Fb`, `region=c%23d`, `region=g+h`, `region=*-._`) are present, and that the natively written table reads back correctly through both Comet's native scan and iceberg-java's reader. The existing Iceberg suites (native scan, write action, write detection, rewrite actions, fuzz, reflection, proto translation) were run locally against the new pin on the default Spark profile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
