andygrove opened a new issue, #6139: URL: https://github.com/apache/datafusion-comet/issues/6139
### Describe the bug The native Iceberg write gate admits `s3` / `s3a` data locations without checking whether the S3 settings in effect can be honoured by the native writer. Settings it cannot honour are dropped silently. What reaches the native writer: - From the Hadoop configuration, only six `fs.s3a.*` settings are translated: access key, secret key, session token, endpoint, region and path-style access (`NativeConfig.scala:62-68`, via `hadoopToIcebergS3Properties` at `CometIcebergNativeWrite.scala:684-687`). - FileIO properties are forwarded as-is (`CometIcebergNativeWrite.scala:594-595`), and `load_file_io` passes on every `s3.` / `client.` key (`iceberg_common.rs:39`, `:122`). iceberg-rust ignores keys it does not know. - Credentials otherwise come from Comet's own credential-provider bridge when one is configured, or from opendal's default chain. Settings that are silently lost include: - Hadoop S3A: `fs.s3a.aws.credentials.provider`, `fs.s3a.encryption.algorithm` / `fs.s3a.encryption.key` (SSE-KMS, SSE-C), `fs.s3a.assumed.role.*`, and EMRFS settings for `s3://` - S3FileIO: `client.factory`, `client.credentials-provider`, `s3.remote-signing-enabled` (REST catalogs), `s3.acl`, `s3.write.tags.*`, `s3.write.storage-class`, access-point mappings Two outcomes, depending on the setting: 1. A write that works with iceberg-java fails with an access error, because the native writer uses different credentials. 2. The write succeeds, but the objects are missing the encryption, ACL, tags or storage class the table owner configured. Nothing reports this. The `gs://` path already fails closed for this reason (#5637). S3 does not. ### Steps to reproduce Configure an S3A write that needs a non-default credentials provider or SSE-KMS through `fs.s3a.encryption.*`, enable `spark.comet.iceberg.write.enabled`, and insert into an Iceberg table on `s3a://`. The plan shows `CometIcebergWriteExec`, and the resulting objects are written without the configured encryption (or the write fails on credentials). ### Expected behavior Fail closed. The gate should decline the native write, with a reason, when any Hadoop `fs.s3a.*` or FileIO `s3.*` / `client.*` setting outside an allow-list of settings the native writer honours is set. Settings could then be added to the allow-list one at a time as the native writer gains support. ### Additional context Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5643 (keep-or-lift per eligibility restriction), #5637. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
