andygrove opened a new pull request, #6305: URL: https://github.com/apache/datafusion-comet/pull/6305
Backport of #6216 to `branch-1.1`. Cherry-picked from `31b38196dbeeb2e976ac9a654fa6bec771c582e9` without conflicts. The six files it changes are identical on `branch-1.1` and on `main` just before #6216, so the diff is byte-identical to upstream. ## Which issue does this PR close? Closes #6144 on `branch-1.1`. #6216 already closed it on `main`. ## Rationale for this change #6216 merged after `branch-1.1` was cut at 36ab57c68, so 1.1.0 would ship the gap described in #6144. The native Iceberg writer always places data files at `<data location>/<partition path>/<file>` and never consults the table's `LocationProvider`. The eligibility gate declines a custom provider only when `write.location-provider.impl` is set, but a custom `TableOperations` can supply one without that property. The gate then admits the write, and the native writer lays the files out differently from iceberg-java. This only affects users who enable the native writer (`spark.comet.iceberg.write.enabled`), which is experimental and off by default in 1.1.0. ## What changes are included in this PR? The original change, so see #6216 for the details. No adaptations were needed. In short: - `IcebergReflection` resolves `table.locationProvider()`. - `CometIcebergNativeWrite` admits a native write only when that provider is Iceberg's built-in `DefaultLocationProvider`, and the fallback reason names the provider class. - The user guide's and contributor guide's Iceberg write pages list the check. ## How are these changes tested? The original PR's tests, run locally on `branch-1.1` with the default profile (Spark 4.1, Scala 2.13, JDK 17): - `IcebergReflectionSuite` and `CometIcebergWriteDetectionSuite` pass, 78 tests, the same count #6216 reported. - With `CometIcebergNativeWrite.scala` reverted to `branch-1.1`'s copy, the new `fall-back: custom TableOperations LocationProvider that no property reveals` fails with `expected Unsupported for custom_location_provider, got Compatible(None,None)`. So without this PR, `branch-1.1` hands that write to the native writer. - `prettier --check` passes on the two changed guides. Against `branch-1.1`, the changed paths route this pull request to every suite except Spark 3.4's SQL job and the benchmark check. That includes every Spark profile, macOS and all four Iceberg versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
