Hi, Ingo I was looking into the aws dependeencies, and from https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html the minimum required version to use the feature is 1.11.704.
So 1.11.788 should be sufficient? Can you point it to me where it says that 1.11.951 is the minimum requirement? Thank you. Rommel On Sun, Sep 26, 2021 at 12:34 AM Ingo Bürk <i...@ververica.com> wrote: > Hi Thomas, > > I haven't encountered that before, sorry. I assume you're still using > Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum > required versions to use this feature, so I'm not sure just setting the > credentials provider alone would be sufficient. The PR was currently only > merged for the upcoming 1.14 release, but perhaps you could try the current > release candidate to see if it works with that? If that works we could also > think about backporting this change, we just initially didn't do that since > upgrading those dependencies has a certain operational risk and we want to > wait for user feedback first. > > > Best > Ingo > > On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang <w...@datability.io> wrote: > >> Ingo, >> >> I dig into the Flink code a little bit. It looks like the key for >> specifying the roleArn and roleSessionName are >> fs.s3a.aws.credentials.provider: >> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >> fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... >> fs.s3a.aws.credentials.provider.role.sessionName: ... >> >> However, for some reason, I'm still getting the same error. Please help! >> Thanks. >> >> Thomas >> >> >> On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang <w...@datability.io> wrote: >> >>> Ingo, >>> >>> It looks like I'm now seeing "Caused by: java.lang.NullPointerException: >>> You must specify a value for roleArn and roleSessionName". I assume I would >>> also need to specify that through the configuration file. Could you suggest >>> the key for this configuration? Thanks. >>> >>> Thomas >>> >>> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote: >>> >>>> Thanks Ingo. Adding the following setting worked. >>>> >>>> fs.s3a.aws.credentials.provider: >>>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >>>> >>>> Thomas >>>> >>>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <i...@ververica.com> wrote: >>>> >>>>> Hi Thomas, >>>>> >>>>> I think you might be looking for this: >>>>> https://github.com/apache/flink/pull/16717 >>>>> >>>>> >>>>> Best >>>>> Ingo >>>>> >>>>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm using the official docker image: >>>>>> apache/flink:1.12.1-scala_2.11-java11 >>>>>> >>>>>> I'm trying to run a Flink job on an EKS cluster. The job is running >>>>>> under a k8s service account that is tied to an IAM role. If I'm not using >>>>>> s3 as RocksDB checkpoint backend, everything works just fine. However, >>>>>> when >>>>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied. >>>>>> >>>>>> The IAM role tied to the service account has the appropriate >>>>>> permissions to s3. However the underlying role tied to the EKS node >>>>>> doesn't. After debugging with AWS support, it looks like the request to >>>>>> s3 >>>>>> was made under the EKS node role, not the role tied to the service >>>>>> account. >>>>>> Thus the permission denial. >>>>>> >>>>>> With the same Flink application, I'm also making requests to AWS >>>>>> Secrets Manager to get some sensitive information and those requests were >>>>>> made explicitly with AWS Java SDK 2.x bundled in the same application Jar >>>>>> file. Those requests were made correctly with the IAM role tied to the >>>>>> service account. >>>>>> >>>>>> Based on the info above, I suspect Flink may be using an older >>>>>> version of the AWS SDK that doesn't support assuming an IAM role via an >>>>>> IODC web identity token file. Please see AWS doc here: >>>>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >>>>>> >>>>>> Could someone help me confirm this bug and maybe have it fixed some >>>>>> time? Thanks. >>>>>> >>>>>> Thomas >>>>>> >>>>> -- Yours Rommel ************************************* I waited patiently for the LORD; he turned to me and heard my cry. He lifted me out of the slimy pit, out of the mud and mire; he set my feet on a rock and gave me a firm place to stand. *************************************
