Hi Thomas, I haven't encountered that before, sorry. I assume you're still using Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum required versions to use this feature, so I'm not sure just setting the credentials provider alone would be sufficient. The PR was currently only merged for the upcoming 1.14 release, but perhaps you could try the current release candidate to see if it works with that? If that works we could also think about backporting this change, we just initially didn't do that since upgrading those dependencies has a certain operational risk and we want to wait for user feedback first.
Best Ingo On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang <w...@datability.io> wrote: > Ingo, > > I dig into the Flink code a little bit. It looks like the key for > specifying the roleArn and roleSessionName are > fs.s3a.aws.credentials.provider: > com.amazonaws.auth.WebIdentityTokenCredentialsProvider > fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/... > fs.s3a.aws.credentials.provider.role.sessionName: ... > > However, for some reason, I'm still getting the same error. Please help! > Thanks. > > Thomas > > > On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang <w...@datability.io> wrote: > >> Ingo, >> >> It looks like I'm now seeing "Caused by: java.lang.NullPointerException: >> You must specify a value for roleArn and roleSessionName". I assume I would >> also need to specify that through the configuration file. Could you suggest >> the key for this configuration? Thanks. >> >> Thomas >> >> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote: >> >>> Thanks Ingo. Adding the following setting worked. >>> >>> fs.s3a.aws.credentials.provider: >>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider >>> >>> Thomas >>> >>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <i...@ververica.com> wrote: >>> >>>> Hi Thomas, >>>> >>>> I think you might be looking for this: >>>> https://github.com/apache/flink/pull/16717 >>>> >>>> >>>> Best >>>> Ingo >>>> >>>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm using the official docker image: >>>>> apache/flink:1.12.1-scala_2.11-java11 >>>>> >>>>> I'm trying to run a Flink job on an EKS cluster. The job is running >>>>> under a k8s service account that is tied to an IAM role. If I'm not using >>>>> s3 as RocksDB checkpoint backend, everything works just fine. However, >>>>> when >>>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied. >>>>> >>>>> The IAM role tied to the service account has the appropriate >>>>> permissions to s3. However the underlying role tied to the EKS node >>>>> doesn't. After debugging with AWS support, it looks like the request to s3 >>>>> was made under the EKS node role, not the role tied to the service >>>>> account. >>>>> Thus the permission denial. >>>>> >>>>> With the same Flink application, I'm also making requests to AWS >>>>> Secrets Manager to get some sensitive information and those requests were >>>>> made explicitly with AWS Java SDK 2.x bundled in the same application Jar >>>>> file. Those requests were made correctly with the IAM role tied to the >>>>> service account. >>>>> >>>>> Based on the info above, I suspect Flink may be using an older version >>>>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web >>>>> identity token file. Please see AWS doc here: >>>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >>>>> >>>>> Could someone help me confirm this bug and maybe have it fixed some >>>>> time? Thanks. >>>>> >>>>> Thomas >>>>> >>>>
