[ https://issues.apache.org/jira/browse/HADOOP-19535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Syed Shameerur Rahman updated HADOOP-19535: ------------------------------------------- Description: The current default s3 credential provider chain is set in the order of {code:java} org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code} Refer [code ref |https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for more details. This works perfectly fine when used in AWS EC2, EMR Serverless, but not with AWS EKS pods. For EKS pods, It is recommended to use {code:java} software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider , software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider (PodIdentity is enabled){code} WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that enables applications to obtain temporary AWS credentials by assuming an IAM role using a web identity token (like OAuth or OIDC tokens). It's particularly important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles for Service Accounts) work. ContainerCredentialsProvider is already part of org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider was: The current default s3 credential provider chain is set in the order of {code:java} org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code} Refer [code ref |https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for more details. This works perfectly fine when used in AWS EC2, EMR Serverless, but not with AWS EKS pods. For EKS pods, It is recommended to use {code:java} software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider , software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider{code} WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that enables applications to obtain temporary AWS credentials by assuming an IAM role using a web identity token (like OAuth or OIDC tokens). It's particularly important in EKS as it's the underlying mechanism that makes IRSA (IAM Roles for Service Accounts) work. ContainerCredentialsProvider is already part of org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider > S3A : Add WebIdentityTokenFileCredentialsProvider to default S3 credential > provider chain > ----------------------------------------------------------------------------------------- > > Key: HADOOP-19535 > URL: https://issues.apache.org/jira/browse/HADOOP-19535 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 > Reporter: Syed Shameerur Rahman > Assignee: Syed Shameerur Rahman > Priority: Major > > The current default s3 credential provider chain is set in the order of > {code:java} > org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider{code} > Refer [code ref > |https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1450]for > more details. > > This works perfectly fine when used in AWS EC2, EMR Serverless, but not with > AWS EKS pods. > > For EKS pods, It is recommended to use > {code:java} > software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider > , software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider > (PodIdentity is enabled){code} > WebIdentityTokenFileCredentialsProvider is an AWS credentials provider that > enables applications to obtain temporary AWS credentials by assuming an IAM > role using a web identity token (like OAuth or OIDC tokens). It's > particularly important in EKS as it's the underlying mechanism that makes > IRSA (IAM Roles for Service Accounts) work. > > > ContainerCredentialsProvider is already part of > org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org