[ 
https://issues.apache.org/jira/browse/HADOOP-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Syed Shameerur Rahman updated HADOOP-19535:
-------------------------------------------
    Description: 
The current default s3 credential provider chain is set in the order of 
{code:java}
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
Refer [code ref 
|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
 more details.

 

This works perfectly fine when used in AWS EC2, EMR Serverless, but not with 
AWS EKS pods.

 

For EKS pods, It is recommended to use
{code:java}
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider 
, software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider 
(PodIdentity is enabled){code}
WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that 
enables applications to obtain temporary AWS credentials by assuming an IAM 
role using a web identity token (like OAuth or OIDC tokens). It's particularly 
important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles 
for Service Accounts) work.

 

 

ContainerCredentialsProvider is already part of 
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

  was:
The current default s3 credential provider chain is set in the order of 
{code:java}
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
Refer [code ref 
|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
 more details.

 

This works perfectly fine when used in AWS EC2, EMR Serverless, but not with 
AWS EKS pods.

 

For EKS pods, It is recommended to use
{code:java}
software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider 
, software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider{code}
WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that 
enables applications to obtain temporary AWS credentials by assuming an IAM 
role using a web identity token (like OAuth or OIDC tokens). It's particularly 
important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles 
for Service Accounts) work.

 

 

ContainerCredentialsProvider is already part of 
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider


> S3A : Add WebIdentityTokenFileCredentialsProvider to default S3 credential 
> provider chain
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-19535
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19535
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Syed Shameerur Rahman
>            Assignee: Syed Shameerur Rahman
>            Priority: Major
>
> The current default s3 credential provider chain is set in the order of 
> {code:java}
> org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code}
> Refer [code ref 
> |https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for
>  more details.
>  
> This works perfectly fine when used in AWS EC2, EMR Serverless, but not with 
> AWS EKS pods.
>  
> For EKS pods, It is recommended to use
> {code:java}
> software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider
>  , software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider 
> (PodIdentity is enabled){code}
> WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that 
> enables applications to obtain temporary AWS credentials by assuming an IAM 
> role using a web identity token (like OAuth or OIDC tokens). It's 
> particularly important in EKS as it's the underlying mechanism that makes 
> IRSA (IAM Roles for Service Accounts) work.
>  
>  
> ContainerCredentialsProvider is already part of 
> org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to