unikdahal opened a new issue, #5636: URL: https://github.com/apache/datafusion-comet/issues/5636
### Describe the bug The native Iceberg write path currently pins iceberg-rust revision: `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7` This revision predates apache/iceberg-rust#2875, which fixed `PartitionSpec::partition_to_path` to URL-encode partition field names and values. As a result, Comet's native Iceberg writer can generate unescaped partition paths for values containing special characters such as `#`, `?`, or `/`. This is not only a directory-layout difference. For example, an S3 partition value containing `#` can produce a committed location such as: ```text s3://bucket/table/data/p=a#b/file.parquet ``` Iceberg Java's `S3FileIO` treats `#` and `?` as URI fragment/query delimiters when resolving the object key, so a location produced by the older iceberg-rust behavior can resolve to a different key when later read through the standard Iceberg Java path. Upstream iceberg-rust already fixed this in: * apache/iceberg-rust#2874 * apache/iceberg-rust#2875 ### Steps to reproduce _No response_ ### Expected behavior Native Iceberg writes should generate partition paths compatible with Iceberg Java's `PartitionSpec.partitionToPath` behavior. Comet should consume an iceberg-rust revision containing apache/iceberg-rust#2875 and include a native-write → JVM-read interoperability regression test for escaped partition paths. ### Additional context This was found during follow-up review of #5361. The upstream issue has already been fixed, so no new iceberg-rust issue is required. The remaining work is for Comet to update its pinned iceberg-rust revision and verify interoperability. Upstream fix: * apache/iceberg-rust#2874 * apache/iceberg-rust#2875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
