jzhuge opened a new issue, #15229:
URL: https://github.com/apache/iceberg/issues/15229
## Current Behavior
When both `client.credentials-provider` (explicit custom provider) and
`client.refresh-credentials-endpoint` are configured, the explicit custom
provider is **ignored** and `VendedCredentialsProvider` is used instead.
**Affected code:**
`aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java:215-236`
## Current Priority in credentialsProvider()
```java
public AwsCredentialsProvider credentialsProvider(
String accessKeyId, String secretAccessKey, String sessionToken) {
// 1. Implicit vended credentials checked FIRST
if (refreshCredentialsEnabled &&
!Strings.isNullOrEmpty(refreshCredentialsEndpoint)) {
return credentialsProvider(VendedCredentialsProvider.class.getName());
}
// 2. Static credentials
if (!Strings.isNullOrEmpty(accessKeyId) &&
!Strings.isNullOrEmpty(secretAccessKey)) {
return StaticCredentialsProvider.create(...);
}
// 3. Custom provider checked LAST (too late!)
if (!Strings.isNullOrEmpty(this.clientCredentialsProvider)) {
return credentialsProvider(this.clientCredentialsProvider);
}
// 4. Default
return DefaultCredentialsProvider.builder().build();
}
```
## Problem
Users cannot override `VendedCredentialsProvider` with a custom
implementation even when explicitly configured:
```properties
# User explicitly sets custom provider
client.credentials-provider=com.example.CustomCredentialsProvider
# But this triggers implicit VendedCredentialsProvider, ignoring the above
client.refresh-credentials-endpoint=https://catalog/credentials
# Result: CustomCredentialsProvider is never used!
```
## Expected Behavior
**Explicit configuration should take precedence over implicit behavior.**
When `client.credentials-provider` is set, it should be used regardless of
whether `refresh-credentials-endpoint` is configured.
## Proposed Fix
Reorder the priority to check custom provider BEFORE implicit vended
credentials:
```java
public AwsCredentialsProvider credentialsProvider(
String accessKeyId, String secretAccessKey, String sessionToken) {
// 1. Static credentials (highest priority for inline config)
if (!Strings.isNullOrEmpty(accessKeyId) &&
!Strings.isNullOrEmpty(secretAccessKey)) {
return StaticCredentialsProvider.create(...);
}
// 2. Custom provider (explicit user choice)
if (!Strings.isNullOrEmpty(this.clientCredentialsProvider)) {
clientCredentialsProviderProperties.put(
VendedCredentialsProvider.URI, refreshCredentialsEndpoint);
return credentialsProvider(this.clientCredentialsProvider);
}
// 3. Implicit vended credentials (fallback when no custom provider)
if (refreshCredentialsEnabled &&
!Strings.isNullOrEmpty(refreshCredentialsEndpoint)) {
clientCredentialsProviderProperties.put(
VendedCredentialsProvider.URI, refreshCredentialsEndpoint);
return credentialsProvider(VendedCredentialsProvider.class.getName());
}
// 4. Default
return DefaultCredentialsProvider.builder().build();
}
```
## Use Case
At Netflix, we're implementing distributed credential vending for Spark
executors. Executors need to use a custom provider that fetches credentials
from the driver via RPC (to avoid N+1 requests to the authorization server),
while the driver uses the standard `VendedCredentialsProvider` to fetch from
the REST catalog.
This requires:
- **Driver:**
`client.credentials-provider=org.apache.iceberg.aws.s3.VendedCredentialsProvider`
- **Executors:**
`client.credentials-provider=com.netflix.iceberg.security.VendedCredsProvider`
(RPC wrapper)
Both need `client.refresh-credentials-endpoint` set (for the underlying
credential fetch), but currently the explicit provider config is ignored.
## Impact
- **Breaking change:** No - both code paths remain, only priority changes
- **Backward compatibility:** Yes - existing implicit users unaffected
- **User benefit:** Explicit configuration works as expected
## Additional Context
This issue affects any user who wants to:
- Wrap `VendedCredentialsProvider` with custom logic (caching, metrics, etc.)
- Use a different credential vending implementation while still setting the
endpoint
- Override implicit behavior with explicit configuration
The fix is minimal (7 lines reordered) and follows the principle: **explicit
> implicit**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
