Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Hi, Ingo I was looking into the aws dependeencies, and from https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html the minimum required version to use the feature is 1.11.704. So 1.11.788 should be sufficient? Can you point it to me where it says that 1.11.951 is the minimum requirement? Thank you. Rommel On Sun, Sep 26, 2021 at 12:34 AM Ingo Bürk wrote: > Hi Thomas, > > I haven't encountered that before, sorry. I assume you're still using > Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum > required versions to use this feature, so I'm not sure just setting the > credentials provider alone would be sufficient. The PR was currently only > merged for the upcoming 1.14 release, but perhaps you could try the current > release candidate to see if it works with that? If that works we could also > think about backporting this change, we just initially didn't do that since > upgrading those dependencies has a certain operational risk and we want to > wait for user feedback first. > > > Best > Ingo > > On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang wrote: > >> Ingo, >> >> I dig into the Flink code a little bit. It looks like the key for >> specifying the roleArn and roleSessionName are >> fs.s3a.aws.credentials.provider: >> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >> fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... >> fs.s3a.aws.credentials.provider.role.sessionName: ... >> >> However, for some reason, I'm still getting the same error. Please help! >> Thanks. >> >> Thomas >> >> >> On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang wrote: >> >>> Ingo, >>> >>> It looks like I'm now seeing "Caused by: java.lang.NullPointerException: >>> You must specify a value for roleArn and roleSessionName". I assume I would >>> also need to specify that through the configuration file. Could you suggest >>> the key for this configuration? Thanks. >>> >>> Thomas >>> >>> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang wrote: >>> Thanks Ingo. Adding the following setting worked. fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider Thomas On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717 > > > Best > Ingo > > On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > >> Hi, >> >> I'm using the official docker image: >> apache/flink:1.12.1-scala_2.11-java11 >> >> I'm trying to run a Flink job on an EKS cluster. The job is running >> under a k8s service account that is tied to an IAM role. If I'm not using >> s3 as RocksDB checkpoint backend, everything works just fine. However, >> when >> I enabled s3 as RocksDB checkpoint backend, I got permission denied. >> >> The IAM role tied to the service account has the appropriate >> permissions to s3. However the underlying role tied to the EKS node >> doesn't. After debugging with AWS support, it looks like the request to >> s3 >> was made under the EKS node role, not the role tied to the service >> account. >> Thus the permission denial. >> >> With the same Flink application, I'm also making requests to AWS >> Secrets Manager to get some sensitive information and those requests were >> made explicitly with AWS Java SDK 2.x bundled in the same application Jar >> file. Those requests were made correctly with the IAM role tied to the >> service account. >> >> Based on the info above, I suspect Flink may be using an older >> version of the AWS SDK that doesn't support assuming an IAM role via an >> IODC web identity token file. Please see AWS doc here: >> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >> >> Could someone help me confirm this bug and maybe have it fixed some >> time? Thanks. >> >> Thomas >> > -- Yours Rommel * I waited patiently for the LORD; he turned to me and heard my cry. He lifted me out of the slimy pit, out of the mud and mire; he set my feet on a rock and gave me a firm place to stand. *
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Hi Thomas, I haven't encountered that before, sorry. I assume you're still using Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum required versions to use this feature, so I'm not sure just setting the credentials provider alone would be sufficient. The PR was currently only merged for the upcoming 1.14 release, but perhaps you could try the current release candidate to see if it works with that? If that works we could also think about backporting this change, we just initially didn't do that since upgrading those dependencies has a certain operational risk and we want to wait for user feedback first. Best Ingo On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang wrote: > Ingo, > > I dig into the Flink code a little bit. It looks like the key for > specifying the roleArn and roleSessionName are > fs.s3a.aws.credentials.provider: > com.amazonaws.auth.WebIdentityTokenCredentialsProvider > fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... > fs.s3a.aws.credentials.provider.role.sessionName: ... > > However, for some reason, I'm still getting the same error. Please help! > Thanks. > > Thomas > > > On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang wrote: > >> Ingo, >> >> It looks like I'm now seeing "Caused by: java.lang.NullPointerException: >> You must specify a value for roleArn and roleSessionName". I assume I would >> also need to specify that through the configuration file. Could you suggest >> the key for this configuration? Thanks. >> >> Thomas >> >> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang wrote: >> >>> Thanks Ingo. Adding the following setting worked. >>> >>> fs.s3a.aws.credentials.provider: >>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >>> >>> Thomas >>> >>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: >>> Hi Thomas, I think you might be looking for this: https://github.com/apache/flink/pull/16717 Best Ingo On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > Hi, > > I'm using the official docker image: > apache/flink:1.12.1-scala_2.11-java11 > > I'm trying to run a Flink job on an EKS cluster. The job is running > under a k8s service account that is tied to an IAM role. If I'm not using > s3 as RocksDB checkpoint backend, everything works just fine. However, > when > I enabled s3 as RocksDB checkpoint backend, I got permission denied. > > The IAM role tied to the service account has the appropriate > permissions to s3. However the underlying role tied to the EKS node > doesn't. After debugging with AWS support, it looks like the request to s3 > was made under the EKS node role, not the role tied to the service > account. > Thus the permission denial. > > With the same Flink application, I'm also making requests to AWS > Secrets Manager to get some sensitive information and those requests were > made explicitly with AWS Java SDK 2.x bundled in the same application Jar > file. Those requests were made correctly with the IAM role tied to the > service account. > > Based on the info above, I suspect Flink may be using an older version > of the AWS SDK that doesn't support assuming an IAM role via an IODC web > identity token file. Please see AWS doc here: > https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html > > Could someone help me confirm this bug and maybe have it fixed some > time? Thanks. > > Thomas >
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Ingo, I dig into the Flink code a little bit. It looks like the key for specifying the roleArn and roleSessionName are fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... fs.s3a.aws.credentials.provider.role.sessionName: ... However, for some reason, I'm still getting the same error. Please help! Thanks. Thomas On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang wrote: > Ingo, > > It looks like I'm now seeing "Caused by: java.lang.NullPointerException: > You must specify a value for roleArn and roleSessionName". I assume I would > also need to specify that through the configuration file. Could you suggest > the key for this configuration? Thanks. > > Thomas > > On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang wrote: > >> Thanks Ingo. Adding the following setting worked. >> >> fs.s3a.aws.credentials.provider: >> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >> >> Thomas >> >> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: >> >>> Hi Thomas, >>> >>> I think you might be looking for this: >>> https://github.com/apache/flink/pull/16717 >>> >>> >>> Best >>> Ingo >>> >>> On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: >>> Hi, I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 I'm trying to run a Flink job on an EKS cluster. The job is running under a k8s service account that is tied to an IAM role. If I'm not using s3 as RocksDB checkpoint backend, everything works just fine. However, when I enabled s3 as RocksDB checkpoint backend, I got permission denied. The IAM role tied to the service account has the appropriate permissions to s3. However the underlying role tied to the EKS node doesn't. After debugging with AWS support, it looks like the request to s3 was made under the EKS node role, not the role tied to the service account. Thus the permission denial. With the same Flink application, I'm also making requests to AWS Secrets Manager to get some sensitive information and those requests were made explicitly with AWS Java SDK 2.x bundled in the same application Jar file. Those requests were made correctly with the IAM role tied to the service account. Based on the info above, I suspect Flink may be using an older version of the AWS SDK that doesn't support assuming an IAM role via an IODC web identity token file. Please see AWS doc here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html Could someone help me confirm this bug and maybe have it fixed some time? Thanks. Thomas >>>
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Ingo, It looks like I'm now seeing "Caused by: java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName". I assume I would also need to specify that through the configuration file. Could you suggest the key for this configuration? Thanks. Thomas On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang wrote: > Thanks Ingo. Adding the following setting worked. > > fs.s3a.aws.credentials.provider: > com.amazonaws.auth.WebIdentityTokenCredentialsProvider > > Thomas > > On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: > >> Hi Thomas, >> >> I think you might be looking for this: >> https://github.com/apache/flink/pull/16717 >> >> >> Best >> Ingo >> >> On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: >> >>> Hi, >>> >>> I'm using the official docker image: >>> apache/flink:1.12.1-scala_2.11-java11 >>> >>> I'm trying to run a Flink job on an EKS cluster. The job is running >>> under a k8s service account that is tied to an IAM role. If I'm not using >>> s3 as RocksDB checkpoint backend, everything works just fine. However, when >>> I enabled s3 as RocksDB checkpoint backend, I got permission denied. >>> >>> The IAM role tied to the service account has the appropriate permissions >>> to s3. However the underlying role tied to the EKS node doesn't. After >>> debugging with AWS support, it looks like the request to s3 was made under >>> the EKS node role, not the role tied to the service account. Thus the >>> permission denial. >>> >>> With the same Flink application, I'm also making requests to AWS Secrets >>> Manager to get some sensitive information and those requests were made >>> explicitly with AWS Java SDK 2.x bundled in the same application Jar file. >>> Those requests were made correctly with the IAM role tied to the service >>> account. >>> >>> Based on the info above, I suspect Flink may be using an older version >>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web >>> identity token file. Please see AWS doc here: >>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >>> >>> Could someone help me confirm this bug and maybe have it fixed some >>> time? Thanks. >>> >>> Thomas >>> >>
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Thanks Ingo. Adding the following setting worked. fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider Thomas On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717 > > > Best > Ingo > > On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > >> Hi, >> >> I'm using the official docker image: >> apache/flink:1.12.1-scala_2.11-java11 >> >> I'm trying to run a Flink job on an EKS cluster. The job is running under >> a k8s service account that is tied to an IAM role. If I'm not using s3 as >> RocksDB checkpoint backend, everything works just fine. However, when I >> enabled s3 as RocksDB checkpoint backend, I got permission denied. >> >> The IAM role tied to the service account has the appropriate permissions >> to s3. However the underlying role tied to the EKS node doesn't. After >> debugging with AWS support, it looks like the request to s3 was made under >> the EKS node role, not the role tied to the service account. Thus the >> permission denial. >> >> With the same Flink application, I'm also making requests to AWS Secrets >> Manager to get some sensitive information and those requests were made >> explicitly with AWS Java SDK 2.x bundled in the same application Jar file. >> Those requests were made correctly with the IAM role tied to the service >> account. >> >> Based on the info above, I suspect Flink may be using an older version of >> the AWS SDK that doesn't support assuming an IAM role via an IODC web >> identity token file. Please see AWS doc here: >> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >> >> Could someone help me confirm this bug and maybe have it fixed some time? >> Thanks. >> >> Thomas >> >
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
We need to overwrite using WebIdentityTokenFileCredentialsProviderhttps://github.com/aws/aws-sdk-java-v2/issues/1470#issuecomment-543601232. otherwise java takes presidency to secret key and access keys than SA On Saturday, September 25, 2021, 04:37:22 PM EDT, Xiangyu Su wrote: Hi Thomas,did you try to login to EKS node and run some aws command like : aws s3 ls ?It sounds like EKS issue, but not 100% sure.Best On Sat, 25 Sept 2021 at 22:12, Ingo Bürk wrote: Hi Thomas, I think you might be looking for this: https://github.com/apache/flink/pull/16717 BestIngo On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: Hi, I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 I'm trying to run a Flink job on an EKS cluster. The job is running under a k8s service account that is tied to an IAM role. If I'm not using s3 as RocksDB checkpoint backend, everything works just fine. However, when I enabled s3 as RocksDB checkpoint backend, I got permission denied. The IAM role tied to the service account has the appropriate permissions to s3. However the underlying role tied to the EKS node doesn't. After debugging with AWS support, it looks like the request to s3 was made under the EKS node role, not the role tied to the service account. Thus the permission denial. With the same Flink application, I'm also making requests to AWS Secrets Manager to get some sensitive information and those requests were made explicitly with AWS Java SDK 2.x bundled in the same application Jar file. Those requests were made correctly with the IAM role tied to the service account. Based on the info above, I suspect Flink may be using an older version of the AWS SDK that doesn't support assuming an IAM role via an IODC web identity token file. Please see AWS doc here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html Could someone help me confirm this bug and maybe have it fixed some time? Thanks. Thomas -- Xiangyu Su Java Developer xian...@smaato.com Smaato Inc. San Francisco - New York - Hamburg - Singapore www.smaato.com Germany: Barcastraße 5 22087 Hamburg Germany M 0049(176)43330282 The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Hi Thomas, did you try to login to EKS node and run some aws command like : aws s3 ls ? It sounds like EKS issue, but not 100% sure. Best On Sat, 25 Sept 2021 at 22:12, Ingo Bürk wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717 > > > Best > Ingo > > On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > >> Hi, >> >> I'm using the official docker image: >> apache/flink:1.12.1-scala_2.11-java11 >> >> I'm trying to run a Flink job on an EKS cluster. The job is running under >> a k8s service account that is tied to an IAM role. If I'm not using s3 as >> RocksDB checkpoint backend, everything works just fine. However, when I >> enabled s3 as RocksDB checkpoint backend, I got permission denied. >> >> The IAM role tied to the service account has the appropriate permissions >> to s3. However the underlying role tied to the EKS node doesn't. After >> debugging with AWS support, it looks like the request to s3 was made under >> the EKS node role, not the role tied to the service account. Thus the >> permission denial. >> >> With the same Flink application, I'm also making requests to AWS Secrets >> Manager to get some sensitive information and those requests were made >> explicitly with AWS Java SDK 2.x bundled in the same application Jar file. >> Those requests were made correctly with the IAM role tied to the service >> account. >> >> Based on the info above, I suspect Flink may be using an older version of >> the AWS SDK that doesn't support assuming an IAM role via an IODC web >> identity token file. Please see AWS doc here: >> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >> >> Could someone help me confirm this bug and maybe have it fixed some time? >> Thanks. >> >> Thomas >> > -- Xiangyu Su Java Developer xian...@smaato.com Smaato Inc. San Francisco - New York - Hamburg - Singapore www.smaato.com Germany: Barcastraße 5 22087 Hamburg Germany M 0049(176)43330282 The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Hi Thomas, I think you might be looking for this: https://github.com/apache/flink/pull/16717 Best Ingo On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote: > Hi, > > I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 > > I'm trying to run a Flink job on an EKS cluster. The job is running under > a k8s service account that is tied to an IAM role. If I'm not using s3 as > RocksDB checkpoint backend, everything works just fine. However, when I > enabled s3 as RocksDB checkpoint backend, I got permission denied. > > The IAM role tied to the service account has the appropriate permissions > to s3. However the underlying role tied to the EKS node doesn't. After > debugging with AWS support, it looks like the request to s3 was made under > the EKS node role, not the role tied to the service account. Thus the > permission denial. > > With the same Flink application, I'm also making requests to AWS Secrets > Manager to get some sensitive information and those requests were made > explicitly with AWS Java SDK 2.x bundled in the same application Jar file. > Those requests were made correctly with the IAM role tied to the service > account. > > Based on the info above, I suspect Flink may be using an older version of > the AWS SDK that doesn't support assuming an IAM role via an IODC web > identity token file. Please see AWS doc here: > https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html > > Could someone help me confirm this bug and maybe have it fixed some time? > Thanks. > > Thomas >
Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?
Hi, I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11 I'm trying to run a Flink job on an EKS cluster. The job is running under a k8s service account that is tied to an IAM role. If I'm not using s3 as RocksDB checkpoint backend, everything works just fine. However, when I enabled s3 as RocksDB checkpoint backend, I got permission denied. The IAM role tied to the service account has the appropriate permissions to s3. However the underlying role tied to the EKS node doesn't. After debugging with AWS support, it looks like the request to s3 was made under the EKS node role, not the role tied to the service account. Thus the permission denial. With the same Flink application, I'm also making requests to AWS Secrets Manager to get some sensitive information and those requests were made explicitly with AWS Java SDK 2.x bundled in the same application Jar file. Those requests were made correctly with the IAM role tied to the service account. Based on the info above, I suspect Flink may be using an older version of the AWS SDK that doesn't support assuming an IAM role via an IODC web identity token file. Please see AWS doc here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html Could someone help me confirm this bug and maybe have it fixed some time? Thanks. Thomas
