andygrove opened a new issue, #5645: URL: https://github.com/apache/datafusion-comet/issues/5645
### What is the problem the feature request solves? `native/Cargo.toml` pins `iceberg` and `iceberg-storage-opendal` to a git revision (`3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`) rather than a release. The pin is what the native Iceberg scan and writer are tested against, but nothing decides when it moves, and #5636 is the first bug caused by it drifting behind an upstream fix (partition-path escaping, apache/iceberg-rust#2875). With the writer now producing files that iceberg-java reads back, a stale pin can turn an upstream bug fix into a Comet correctness bug. ### Describe the potential solution Write down and follow a policy, for example: - Bump the pin at least once per Comet release cycle, and immediately when an upstream fix affects file bytes, manifest metadata, or partition layout. - Prefer tagged iceberg-rust releases over arbitrary revisions once a release contains everything Comet needs, so the version is visible in `Cargo.lock` and in the docs. - Each bump runs the Iceberg suites on all four Spark/Iceberg profiles plus the manifest-metrics parity tests, since those are what catch behavior changes in the writer. - Track "waiting on upstream" items (see the eligibility-restrictions issue) so a bump can also lift restrictions. - Note the bump in the release notes, since users reading Comet-written tables with iceberg-java care which writer semantics they got. ### Additional context Part of the native Iceberg writes epic. Related: #5636. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
