snmvaughan opened a new issue, #6207:
URL: https://github.com/apache/datafusion-comet/issues/6207
**Problem.** On the Parquet path,
`object_store::CredentialProvider::get_credential()` receives no request path,
so one S3 store presents one credential and Comet caches one store per bucket
and configuration. A `CometS3CredentialProvider` gets one credential per
bucket, requested with the path of the first file read. Deployments whose
policies differ by location within a bucket, for example one STS session for
`warehouse/sales` and another for `warehouse/finance`, get 403s everywhere but
the first location.
**Proposal.** Add an opt-in `@Public` extension:
```java
public interface CometS3LocationScopedCredentialProvider extends
CometS3CredentialProvider {
List<String> getPolicyLocations(String bucket) throws Exception;
}
```
The provider lists every location in the bucket that has its own policy.
Comet serves each request with the credential of the longest location covering
its path, matched one segment at a time, with the bucket root as an implicit
location. It requests a location's credential by calling
`getCredentialsForPath` with the location as the path. Locations apply to
native Parquet reads; the Iceberg path is unchanged. Providers that implement
only the base interface are unaffected.
**Compatibility.** The change is additive: a new interface with no change to
existing types. Under the versioning policy, adding an abstract method to it
later, or changing how paths match locations, would require a major release.
Implementation: #6031.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
