unikdahal opened a new issue, #5637: URL: https://github.com/apache/datafusion-comet/issues/5637
### Describe the bug The native Iceberg write eligibility path supports `gs://` locations and `NativeConfig.extractObjectStoreOptions` extracts the corresponding `fs.gs.*` settings from the Hadoop configuration. However, `CometIcebergNativeWrite` then passes those extracted settings through: ```scala CometIcebergNativeScan.hadoopToIcebergS3Properties(...) ``` which translates S3/S3A configuration only. For a `gs://` location, the extracted `fs.gs.*` settings are therefore not preserved when building the native GCS FileIO. This is particularly problematic for tables using `HadoopFileIO`, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the Hadoop `Configuration` rather than `FileIO.properties()`. The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration. Depending on the environment, the native writer could: * fail even though the JVM writer succeeds; * use a different GCS endpoint; or * resolve a different ambient/default identity than the one configured for the Hadoop filesystem. Native writes should fail closed rather than silently changing storage identity or endpoint configuration. ### Steps to reproduce _No response_ ### Expected behavior The native writer must either: 1. faithfully translate the relevant Hadoop GCS configuration into the native GCS backend configuration; or 2. fail closed and fall back to the JVM Iceberg writer for `HadoopFileIO + gs://`. Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination. A table using an actual `GCSFileIO` whose properties already provide supported native `gcs.*` settings can remain eligible. ### Additional context Found during follow-up review of #5361. A simple `fs.gs.*` → `gcs.*` key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation. The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
