[
https://issues.apache.org/jira/browse/HADOOP-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HADOOP-19535:
-------------------------------------------
Description:
The current default s3 credential provider chain is set in the order of
{code:java}
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
Refer [code ref
|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
more details.
This works perfectly fine when used in AWS EC2, EMR Serverless, but not with
AWS EKS pods.
For EKS pods, It is recommended to use
{code:java}
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
, software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider
(PodIdentity is enabled){code}
WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that
enables applications to obtain temporary AWS credentials by assuming an IAM
role using a web identity token (like OAuth or OIDC tokens). It's particularly
important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles
for Service Accounts) work.
ContainerCredentialsProvider is already part of
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
was:
The current default s3 credential provider chain is set in the order of
{code:java}
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
Refer [code ref
|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
more details.
This works perfectly fine when used in AWS EC2, EMR Serverless, but not with
AWS EKS pods.
For EKS pods, It is recommended to use
{code:java}
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
, software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider{code}
WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that
enables applications to obtain temporary AWS credentials by assuming an IAM
role using a web identity token (like OAuth or OIDC tokens). It's particularly
important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles
for Service Accounts) work.
ContainerCredentialsProvider is already part of
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> S3A : Add WebIdentityTokenFileCredentialsProvider to default S3 credential
> provider chain
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-19535
> URL: https://issues.apache.org/jira/browse/HADOOP-19535
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Syed Shameerur Rahman
> Assignee: Syed Shameerur Rahman
> Priority: Major
>
> The current default s3 credential provider chain is set in the order of
> {code:java}
> org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
> Refer [code ref
> |https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
> more details.
>
> This works perfectly fine when used in AWS EC2, EMR Serverless, but not with
> AWS EKS pods.
>
> For EKS pods, It is recommended to use
> {code:java}
> software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
> , software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider
> (PodIdentity is enabled){code}
> WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that
> enables applications to obtain temporary AWS credentials by assuming an IAM
> role using a web identity token (like OAuth or OIDC tokens). It's
> particularly important in EKS as it's the underlying mechanism that makes
> IRSA (IAM Roles for Service Accounts) work.
>
>
> ContainerCredentialsProvider is already part of
> org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
