This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5c7110b978c [fix](glue) cache credentials provider in
ConfigurationAWSCredentialsProvider (#65165)
5c7110b978c is described below
commit 5c7110b978c86def1ea60ca7e98e4734d036d414
Author: Calvin Kirs <[email protected]>
AuthorDate: Mon Jul 6 09:52:41 2026 +0800
[fix](glue) cache credentials provider in
ConfigurationAWSCredentialsProvider (#65165)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The AWS SDK signer calls `AWSCredentialsProvider.getCredentials()` for
every Glue API request, but `ConfigurationAWSCredentialsProvider` (used
by the Hive-Glue metastore client via
`aws.catalog.credentials.provider.factory.class`) rebuilt the entire
provider chain — including a fresh `InstanceProfileCredentialsProvider`
/ `STSAssumeRoleSessionCredentialsProvider` — inside every
`getCredentials()` call. The providers' internal credential caching was
therefore defeated, and **every single Glue request triggered a full
IMDS round trip** (token PUT + credential GET under IMDSv2), or a full
STS AssumeRole call in the `glue.role_arn` case.
Low-rate operations survive this, but per-partition fan-out paths
overwhelm the EC2 instance metadata service. Observed in production
(Glue catalog without explicit credentials, base table with ~160k
partitions): `SHOW PARTITIONS` on an MTMV whose base table is a Glue
Hive table runs `MTMVPartitionUtil.isSyncWithPartitions`, which issues
one `GetPartition` RPC per base-table partition; each RPC re-fetched
credentials from IMDS, IMDS throttling then killed one of the fetches
and the whole statement failed:
```
MetaException: Unable to get partition with values: .../2024/01/26/...
com.amazonaws.SdkClientException: Unable to load AWS credentials from any
provider in the chain:
[com.amazonaws.auth.InstanceProfileCredentialsProvider@...: Failed to
connect to service endpoint: ,
EnvironmentVariableCredentialsProvider: ...,
SystemPropertiesCredentialsProvider: ...]
```
Bulk paths (`listPartitionNames`, `SHOW DATABASES`, `table$partitions`)
issue a single RPC and keep working, which makes the failure look
selective and hard to diagnose.
What I changed:
Build the delegate provider once per
`ConfigurationAWSCredentialsProvider` instance (double-checked locking
on a `volatile` field) and let the SDK-native providers cache and
auto-refresh temporary credentials as designed; `refresh()` now forwards
to the delegate. IMDS/STS traffic drops from O(requests) to cold-start
plus periodic refresh.
Notes:
- AK/SK branch behavior is unchanged (now wrapped in
`AWSStaticCredentialsProvider`).
- Config snapshot-at-first-use is safe: `ALTER CATALOG` rebuilds the
HiveConf and the metastore client pool.
- The SDK v2 variant `ConfigurationAWSCredentialsProvider2x` (Iceberg
path) is static-only and not affected.
### Release note
Fix: Glue Hive catalog no longer fetches AWS credentials from IMDS/STS
on every metastore request; fixes 'Unable to load AWS credentials from
any provider in the chain' failures on partition-heavy operations (e.g.
SHOW PARTITIONS on an MTMV over a Glue table).
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [ ] Integration test
- [x] Manual test (add detailed scripts or steps below)
- Verified on a production 26.x deployment: with the fixed jar, MTMV
`SHOW PARTITIONS` over a Glue base table no longer floods IMDS and
completes successfully.
- [ ] No need to test or manual test. Explain why.
---
.../ConfigurationAWSCredentialsProvider.java | 45 +++++++++++++++++-----
1 file changed, 35 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/com/amazonaws/glue/catalog/credentials/ConfigurationAWSCredentialsProvider.java
b/fe/fe-core/src/main/java/com/amazonaws/glue/catalog/credentials/ConfigurationAWSCredentialsProvider.java
index 23ba972fdfe..293e6a099b0 100644
---
a/fe/fe-core/src/main/java/com/amazonaws/glue/catalog/credentials/ConfigurationAWSCredentialsProvider.java
+++
b/fe/fe-core/src/main/java/com/amazonaws/glue/catalog/credentials/ConfigurationAWSCredentialsProvider.java
@@ -20,6 +20,7 @@ package com.amazonaws.glue.catalog.credentials;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
@@ -32,7 +33,13 @@ import org.apache.hadoop.conf.Configuration;
public class ConfigurationAWSCredentialsProvider implements
AWSCredentialsProvider {
- private Configuration conf;
+ private final Configuration conf;
+
+ // The SDK signer invokes getCredentials() on every request, so the
underlying provider must
+ // be built only once: providers like InstanceProfileCredentialsProvider
and
+ // STSAssumeRoleSessionCredentialsProvider cache their temporary
credentials per instance,
+ // and rebuilding them per call would hit IMDS/STS on every single Glue
request.
+ private volatile AWSCredentialsProvider delegate;
public ConfigurationAWSCredentialsProvider(Configuration conf) {
this.conf = conf;
@@ -40,17 +47,34 @@ public class ConfigurationAWSCredentialsProvider implements
AWSCredentialsProvid
@Override
public AWSCredentials getCredentials() {
+ AWSCredentialsProvider provider = delegate;
+ if (provider == null) {
+ synchronized (this) {
+ if (delegate == null) {
+ delegate = buildDelegate();
+ }
+ provider = delegate;
+ }
+ }
+ return provider.getCredentials();
+ }
+
+ private AWSCredentialsProvider buildDelegate() {
String accessKey =
StringUtils.trim(conf.get(AWSGlueConfig.AWS_GLUE_ACCESS_KEY));
String secretKey =
StringUtils.trim(conf.get(AWSGlueConfig.AWS_GLUE_SECRET_KEY));
String sessionToken =
StringUtils.trim(conf.get(AWSGlueConfig.AWS_GLUE_SESSION_TOKEN));
String roleArn =
StringUtils.trim(conf.get(AWSGlueConfig.AWS_GLUE_ROLE_ARN));
String externalId =
StringUtils.trim(conf.get(AWSGlueConfig.AWS_GLUE_EXTERNAL_ID));
if (!StringUtils.isNullOrEmpty(accessKey) &&
!StringUtils.isNullOrEmpty(secretKey)) {
- return (StringUtils.isNullOrEmpty(sessionToken) ? new
BasicAWSCredentials(accessKey,
- secretKey) : new BasicSessionCredentials(accessKey,
secretKey, sessionToken));
+ AWSCredentials credentials =
StringUtils.isNullOrEmpty(sessionToken)
+ ? new BasicAWSCredentials(accessKey, secretKey)
+ : new BasicSessionCredentials(accessKey, secretKey,
sessionToken);
+ return new AWSStaticCredentialsProvider(credentials);
}
- String credentialsProviderModeString =
StringUtils.lowerCase(conf.get(AWSGlueConfig.AWS_CREDENTIALS_PROVIDER_MODE));
- AwsCredentialsProviderMode
credentialsProviderMode=AwsCredentialsProviderMode.fromString(credentialsProviderModeString);
+ String credentialsProviderModeString =
+
StringUtils.lowerCase(conf.get(AWSGlueConfig.AWS_CREDENTIALS_PROVIDER_MODE));
+ AwsCredentialsProviderMode credentialsProviderMode =
+
AwsCredentialsProviderMode.fromString(credentialsProviderModeString);
AWSCredentialsProvider longLivedProvider =
AwsCredentialsProviderFactory.createV1(credentialsProviderMode);
if (!StringUtils.isNullOrEmpty(roleArn)) {
STSAssumeRoleSessionCredentialsProvider.Builder builder =
@@ -60,19 +84,20 @@ public class ConfigurationAWSCredentialsProvider implements
AWSCredentialsProvid
if (!StringUtils.isNullOrEmpty(externalId)) {
builder.withExternalId(externalId);
}
- STSAssumeRoleSessionCredentialsProvider provider = builder.build();
- return provider.getCredentials();
+ return builder.build();
}
if (Config.aws_credentials_provider_version.equalsIgnoreCase("v2")) {
- return longLivedProvider.getCredentials();
+ return longLivedProvider;
}
throw new SdkClientException("Unable to load AWS credentials from any
provider in the chain");
-
}
@Override
public void refresh() {
-
+ AWSCredentialsProvider provider = delegate;
+ if (provider != null) {
+ provider.refresh();
+ }
}
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
