andygrove opened a new pull request, #6603: URL: https://github.com/apache/datafusion-comet/pull/6603
## Which issue does this PR close? None. It documents a behavior change from #6025 in the 1.1.0 upgrade guide. It was part of the draft release notes PR #6469, which is closing because 1.1.0 now fixes nearly every regression that PR listed. ## Rationale for this change Since #6025, which fixed #6024, Comet's native Iceberg scan on EKS with IRSA takes its S3 credentials only from the web-identity role, when the IRSA environment variables are set and the catalog configures no credentials. If that call fails, it no longer falls back to the node role or Pod Identity as 1.0.0 did. So a cluster whose IRSA setup is broken, and that was reading S3 as the node role without anyone noticing, fails native Iceberg reads after upgrading. The 1.1.0 upgrade guide doesn't mention it. ## What changes are included in this PR? A "Native Iceberg Reads on EKS with IRSA" subsection under "Upgrading to Comet 1.1.0". It describes the change, points to the "EKS / IRSA" section of the S3 credential providers page, and names the two ways back: `spark.sql.catalog.<catalog>.s3.comet.credential.webIdentity.enabled=false` for a catalog, and `spark.comet.scan.icebergNative.enabled=false` for a table loaded by path, which has no catalog to set it on. This should go to `branch-1.1` too, because the archived 1.1.0 docs are built from that branch. The commit applies there cleanly. ## How are these changes tested? Docs only. The setting and the page section it names exist on both `main` and `branch-1.1`, and prettier passes on the file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
