Re: Does Spark support role-based authentication and access to Amazon S3? (Kubernetes cluster deployment)

Apologies Koert! Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and

Re: Does Spark support role-based authentication and access to Amazon S3? (Kubernetes cluster deployment)

Hi kurt, I read this document of yours. indeed interesting and pretty recent (9th Dec). I am more focused on GCP and GKE . But obviously the concepts are the same. One thing I noticed, there was a lack of mention of Workload Identity federation

Re: Does Spark support role-based authentication and access to Amazon S3? (Kubernetes cluster deployment)

yes it does using IAM roles for service accounts. see: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html i wrote a little bit about this also here: https://technotes.tresata.com/spark-on-k8s/ On Wed, Dec 13, 2023 at 7:52 AM Atul Patil wrote: > Hello Team, > >

Does Spark support role-based authentication and access to Amazon S3? (Kubernetes cluster deployment)

Hello Team, Does Spark support role-based authentication and access to Amazon S3 for Kubernetes deployment? *Note: we have deployed our spark application in the Kubernetes cluster.* Below are the Hadoop-AWS dependencies we are using: org.apache.hadoop hadoop-aws 3.3.4 We are

Does Spark support role-based authentication and access to Amazon S3? (Kubernetes cluster deployment)

Hello Team, Does Spark support role-based authentication and access to Amazon S3 for Kubernetes deployment? Note: we have deployed our spark application in the Kubernetes cluster. Below are the Hadoop-AWS dependencies we are using: org.apache.hadoop hadoop-aws 3.3.4 We are using the