andygrove commented on PR #6808: URL: https://github.com/apache/datafusion-comet/pull/6808#issuecomment-6087147870
Not a broader list, no. I've only added keys that S3A wouldn't apply to an Iceberg data-file write and that arrive as cluster-wide defaults rather than a per-job choice: Spark 4.1 sets the two committer keys itself whenever `spark-hadoop-cloud` is on the classpath, and the other three are common platform-level tuning. Anything S3A does apply to a write that the native writer never sees (encryption, ACLs, storage class, signing, the credentials provider) has to keep falling back, which is what #6441 is for. I'd rather add a key when someone hits it blocking native writes than guess at the long tail. Is there one you've run into? And yes to the guides. Both name the ignored keys (vectored-read and `downgrade.syncable.exceptions`), so they're stale after this change. I put the update for both in a separate docs-only PR, #6828, to avoid re-running CI here. It should merge after this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
