[jira] [Commented] (HADOOP-17188) Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based credential provider to support use of IRSA on deployments on AWS EKS Cluster
[ https://issues.apache.org/jira/browse/HADOOP-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189791#comment-17189791 ] Arun Ravi M V commented on HADOOP-17188: Here is the Jira ticket and pull request. https://issues.apache.org/jira/browse/SPARK-27872 [https://github.com/apache/spark/pull/24748] > Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based > credential provider to support use of IRSA on deployments on AWS EKS Cluster > - > > Key: HADOOP-17188 > URL: https://issues.apache.org/jira/browse/HADOOP-17188 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Arun Ravi M V >Priority: Minor > > The latest version of AWS SDK has support to use IRSA for providing > credentials to Kubernetes pods which can potentially replace the use of > Kube2IAM. For our Apache Spark on Kubernetes use cases, this feature will be > useful. The current Hadoop AWS component does support adding custom > credential provider but I think if we could add > STSAssumeRoleWithWebIdentitySessionCredentialsProvider support to (using > roleArn, role session name, web Identity Token File) to the hadoop-aws > library, it will be useful for the community as such who use AWS EKS. > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.html] > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder.html > ] > [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17188) Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based credential provider to support use of IRSA on deployments on AWS EKS Cluster
[ https://issues.apache.org/jira/browse/HADOOP-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189283#comment-17189283 ] Steve Loughran commented on HADOOP-17188: - ok, so it was some spark setup change? Is there a SPARK- JIRA to link to? > Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based > credential provider to support use of IRSA on deployments on AWS EKS Cluster > - > > Key: HADOOP-17188 > URL: https://issues.apache.org/jira/browse/HADOOP-17188 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Arun Ravi M V >Priority: Minor > > The latest version of AWS SDK has support to use IRSA for providing > credentials to Kubernetes pods which can potentially replace the use of > Kube2IAM. For our Apache Spark on Kubernetes use cases, this feature will be > useful. The current Hadoop AWS component does support adding custom > credential provider but I think if we could add > STSAssumeRoleWithWebIdentitySessionCredentialsProvider support to (using > roleArn, role session name, web Identity Token File) to the hadoop-aws > library, it will be useful for the community as such who use AWS EKS. > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.html] > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder.html > ] > [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17188) Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based credential provider to support use of IRSA on deployments on AWS EKS Cluster
[ https://issues.apache.org/jira/browse/HADOOP-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188527#comment-17188527 ] Arun Ravi M V commented on HADOOP-17188: We fixed this issue by patching Apache Spark and using com.amazonaws.auth.WebIdentityTokenCredentialsProvider. Details: [https://medium.com/@tunguyen9889/how-to-perform-a-spark-submit-to-amazon-eks-cluster-with-irsa-50af9b26cae] I am marking this issue as will not do as the problem was being propagated from Apache Spark. > Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based > credential provider to support use of IRSA on deployments on AWS EKS Cluster > - > > Key: HADOOP-17188 > URL: https://issues.apache.org/jira/browse/HADOOP-17188 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Arun Ravi M V >Priority: Minor > > The latest version of AWS SDK has support to use IRSA for providing > credentials to Kubernetes pods which can potentially replace the use of > Kube2IAM. For our Apache Spark on Kubernetes use cases, this feature will be > useful. The current Hadoop AWS component does support adding custom > credential provider but I think if we could add > STSAssumeRoleWithWebIdentitySessionCredentialsProvider support to (using > roleArn, role session name, web Identity Token File) to the hadoop-aws > library, it will be useful for the community as such who use AWS EKS. > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.html] > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder.html > ] > [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17188) Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based credential provider to support use of IRSA on deployments on AWS EKS Cluster
[ https://issues.apache.org/jira/browse/HADOOP-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173027#comment-17173027 ] Steve Loughran commented on HADOOP-17188: - If its in the aws SDK JAR we ship -a matter of just listing it on the fs.s3a.credential.provider option * Do this, let us know how it works, and supply docs * we haven't updated the AWS SDK for a while, if that is needed, create a JIRA for that and have a go following the runbook in testing.md * if there are specific changes needed (per-bucket setting of different options..), then yes, a new provider is welcome. Ideally one we can test > Support for AWS STSAssumeRoleWithWebIdentitySessionCredentialsProvider based > credential provider to support use of IRSA on deployments on AWS EKS Cluster > - > > Key: HADOOP-17188 > URL: https://issues.apache.org/jira/browse/HADOOP-17188 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Arun Ravi M V >Priority: Minor > > The latest version of AWS SDK has support to use IRSA for providing > credentials to Kubernetes pods which can potentially replace the use of > Kube2IAM. For our Apache Spark on Kubernetes use cases, this feature will be > useful. The current Hadoop AWS component does support adding custom > credential provider but I think if we could add > STSAssumeRoleWithWebIdentitySessionCredentialsProvider support to (using > roleArn, role session name, web Identity Token File) to the hadoop-aws > library, it will be useful for the community as such who use AWS EKS. > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.html] > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder.html > ] > [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
