IRSA with Flink S3a connector

[Anuj Jain]

Hi,
I have a flink job running on EKS, reading and writing data records to S3
buckets.
I am trying to set up access credentials via AWS IAM.
I followed this:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

I have configured: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
as the credential provider in flink-conf.yaml for hadoop s3a connector, and
annotated my service account with the role.

When running the job, i am getting access denied error
Exception:
Caused by:
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
Not authorized to perform sts:AssumeRoleWithWebIdentity (Service:
AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;
Request ID: 923df33a-802e-47e2-a203-0841aca03dd8; Proxy: null)

I have tried to access S3 buckets from AWS CLI running in a pod with the
same service account and that works.

Am I using the correct credential provider for IAM integration, not sure if
Hadoop S3a supports it.
https://issues.apache.org/jira/browse/HADOOP-18154

Please advise if I am doing anything wrong in setting up credentials via
IAM.

Regards
Anuj Jain

Re: IRSA with Flink S3a connector

[Anuj Jain]

Hi Community,
Looking forward to some advice on the problem.

I also found this similar Jira, but not sure if a fix has been done for the
Hadoop connector - can someone confirm this.
[FLINK-23487] IRSA doesn't work with S3 - ASF JIRA (apache.org)


Is there any other way to integrate Flink source/sink with AWS IAM from EKS
?

Regards
Anuj

On Thu, May 18, 2023 at 12:41 PM Anuj Jain  wrote:

> Hi,
> I have a flink job running on EKS, reading and writing data records to S3
> buckets.
> I am trying to set up access credentials via AWS IAM.
> I followed this:
> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
>
> I have configured: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
> as the credential provider in flink-conf.yaml for hadoop s3a connector, and
> annotated my service account with the role.
>
> When running the job, i am getting access denied error
> Exception:
> Caused by:
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Not authorized to perform sts:AssumeRoleWithWebIdentity (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;
> Request ID: 923df33a-802e-47e2-a203-0841aca03dd8; Proxy: null)
>
> I have tried to access S3 buckets from AWS CLI running in a pod with the
> same service account and that works.
>
> Am I using the correct credential provider for IAM integration, not sure
> if Hadoop S3a supports it.
> https://issues.apache.org/jira/browse/HADOOP-18154
>
> Please advise if I am doing anything wrong in setting up credentials via
> IAM.
>
> Regards
> Anuj Jain
>

Re: IRSA with Flink S3a connector

[Anuj Jain]

Hello,
Please provide some pointers on this issue.

Thanks !!

Regards
Anuj

On Fri, May 19, 2023 at 1:34 PM Anuj Jain  wrote:

> Hi Community,
> Looking forward to some advice on the problem.
>
> I also found this similar Jira, but not sure if a fix has been done for
> the Hadoop connector - can someone confirm this.
> [FLINK-23487] IRSA doesn't work with S3 - ASF JIRA (apache.org)
> 
>
> Is there any other way to integrate Flink source/sink with AWS IAM from
> EKS ?
>
> Regards
> Anuj
>
> On Thu, May 18, 2023 at 12:41 PM Anuj Jain  wrote:
>
>> Hi,
>> I have a flink job running on EKS, reading and writing data records to S3
>> buckets.
>> I am trying to set up access credentials via AWS IAM.
>> I followed this:
>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
>>
>> I have configured: com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>> as the credential provider in flink-conf.yaml for hadoop s3a connector, and
>> annotated my service account with the role.
>>
>> When running the job, i am getting access denied error
>> Exception:
>> Caused by:
>> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
>> Not authorized to perform sts:AssumeRoleWithWebIdentity (Service:
>> AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;
>> Request ID: 923df33a-802e-47e2-a203-0841aca03dd8; Proxy: null)
>>
>> I have tried to access S3 buckets from AWS CLI running in a pod with the
>> same service account and that works.
>>
>> Am I using the correct credential provider for IAM integration, not sure
>> if Hadoop S3a supports it.
>> https://issues.apache.org/jira/browse/HADOOP-18154
>>
>> Please advise if I am doing anything wrong in setting up credentials via
>> IAM.
>>
>> Regards
>> Anuj Jain
>>
>

Re: IRSA with Flink S3a connector

[Martijn Visser]

Hi Anuj,

I recalled another ticket on this topic, which had some things to test. I
don't know if that resolved the issue, can you verify it? See
https://issues.apache.org/jira/browse/FLINK-31095

Best regards,

Martijn

On Tue, May 23, 2023 at 7:04 AM Anuj Jain  wrote:

> Hello,
> Please provide some pointers on this issue.
>
> Thanks !!
>
> Regards
> Anuj
>
> On Fri, May 19, 2023 at 1:34 PM Anuj Jain  wrote:
>
>> Hi Community,
>> Looking forward to some advice on the problem.
>>
>> I also found this similar Jira, but not sure if a fix has been done for
>> the Hadoop connector - can someone confirm this.
>> [FLINK-23487] IRSA doesn't work with S3 - ASF JIRA (apache.org)
>> 
>>
>> Is there any other way to integrate Flink source/sink with AWS IAM from
>> EKS ?
>>
>> Regards
>> Anuj
>>
>> On Thu, May 18, 2023 at 12:41 PM Anuj Jain  wrote:
>>
>>> Hi,
>>> I have a flink job running on EKS, reading and writing data records to
>>> S3 buckets.
>>> I am trying to set up access credentials via AWS IAM.
>>> I followed this:
>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
>>>
>>> I have configured:
>>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider as the credential
>>> provider in flink-conf.yaml for hadoop s3a connector, and annotated my
>>> service account with the role.
>>>
>>> When running the job, i am getting access denied error
>>> Exception:
>>> Caused by:
>>> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
>>> Not authorized to perform sts:AssumeRoleWithWebIdentity (Service:
>>> AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;
>>> Request ID: 923df33a-802e-47e2-a203-0841aca03dd8; Proxy: null)
>>>
>>> I have tried to access S3 buckets from AWS CLI running in a pod with the
>>> same service account and that works.
>>>
>>> Am I using the correct credential provider for IAM integration, not sure
>>> if Hadoop S3a supports it.
>>> https://issues.apache.org/jira/browse/HADOOP-18154
>>>
>>> Please advise if I am doing anything wrong in setting up credentials via
>>> IAM.
>>>
>>> Regards
>>> Anuj Jain
>>>
>>

Re: IRSA with Flink S3a connector

[Anuj Jain]

Hi Martijn,

As I can see FLINK-31095  is
now closed.
I will try the things mentioned there on my test systems and share the
results.

Thanks for your help.

Regards
Anuj

On Tue, May 23, 2023 at 1:10 PM Martijn Visser 
wrote:

> Hi Anuj,
>
> I recalled another ticket on this topic, which had some things to test. I
> don't know if that resolved the issue, can you verify it? See
> https://issues.apache.org/jira/browse/FLINK-31095
>
> Best regards,
>
> Martijn
>
> On Tue, May 23, 2023 at 7:04 AM Anuj Jain  wrote:
>
>> Hello,
>> Please provide some pointers on this issue.
>>
>> Thanks !!
>>
>> Regards
>> Anuj
>>
>> On Fri, May 19, 2023 at 1:34 PM Anuj Jain  wrote:
>>
>>> Hi Community,
>>> Looking forward to some advice on the problem.
>>>
>>> I also found this similar Jira, but not sure if a fix has been done for
>>> the Hadoop connector - can someone confirm this.
>>> [FLINK-23487] IRSA doesn't work with S3 - ASF JIRA (apache.org)
>>> 
>>>
>>> Is there any other way to integrate Flink source/sink with AWS IAM from
>>> EKS ?
>>>
>>> Regards
>>> Anuj
>>>
>>> On Thu, May 18, 2023 at 12:41 PM Anuj Jain  wrote:
>>>
 Hi,
 I have a flink job running on EKS, reading and writing data records to
 S3 buckets.
 I am trying to set up access credentials via AWS IAM.
 I followed this:
 https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

 I have configured:
 com.amazonaws.auth.WebIdentityTokenCredentialsProvider as the credential
 provider in flink-conf.yaml for hadoop s3a connector, and annotated my
 service account with the role.

 When running the job, i am getting access denied error
 Exception:
 Caused by:
 com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
 Not authorized to perform sts:AssumeRoleWithWebIdentity (Service:
 AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;
 Request ID: 923df33a-802e-47e2-a203-0841aca03dd8; Proxy: null)

 I have tried to access S3 buckets from AWS CLI running in a pod with
 the same service account and that works.

 Am I using the correct credential provider for IAM integration, not
 sure if Hadoop S3a supports it.
 https://issues.apache.org/jira/browse/HADOOP-18154

 Please advise if I am doing anything wrong in setting up credentials
 via IAM.

 Regards
 Anuj Jain

>>>