Thanks Alexey and Moritz for your valuable inputs.

We were using ‘STSAssumeRoleSessionCredentialsProvider’ because of AWS SDKv1. 
Now as per response and the issue, we will upgrade AWS SDK and Beam SDK and 
give it a go. Will keep you posted on this in few days time. Thanks.

Regards,
Sankar

From: Moritz Mack <mm...@talend.com>
Reply to: "user@beam.apache.org" <user@beam.apache.org>
Date: Tuesday, 15 November 2022 at 10:45
To: Alexey Romanenko <aromanenko....@gmail.com>, "user@beam.apache.org" 
<user@beam.apache.org>
Subject: Re: Reading from AWS Kinesis Stream Cross account

Hi Sankar,

First, as Alexey pointed out, please try and migrate to the Beam AWS SDK v2 as 
soon as possible. The SDK v1 (including the Kinesis module) has been long 
deprecated and will be removed some time soon.

The AWS API doesn’t support cross-account access for Kinesis using an ARN. This 
is always based on the stream name as shown in the example you’ve linked [1].  
You must use STS / assume role credentials to do this, it can’t be done using 
DefaultAWSCredentialsProviderChain.

Assuming you’ve correctly configured all required policies and roles (on both 
accounts) following [1], you can then use the StsAssumeRoleCredentialsProvider. 
For the Beam AWS SDK v2 this can be done using pipeline options [2], or 
programmatically of course.
Support for StsAssumeRoleCredentialsProvider is more limited in SDK v1, though 
it should also work.

--awsCredentialsProvider={
   "@type": "StsAssumeRoleCredentialsProvider",
   "roleArn": "arn:aws:iam::SOURCE01234567:role/KA-Source-Stream-Role",
   "roleSessionName": "ksassumedrolesession"
}

The STS client will implicitly use the DefaultAWSCredentialsProviderChain to 
assume the above role of the source account using an authenticated principal of 
the sink account (source/sink nomenclature as used in [1]). Please make sure 
your environment is configured accordingly, this part is easy to miss.

Let me know if you have more questions!
Regards,
Moritz


[1] https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html
[2] 
https://beam.apache.org/releases/javadoc/2.42.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html#getAwsCredentialsProvider--

On 14.11.22, 19:05, "Alexey Romanenko" <aromanenko....@gmail.com> wrote:

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 
[1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since 
former is

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO.

PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s 
highly recommended to switch to one based on AWS SDK v2 [2] since former is 
deprecated.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTCweGmqKQ$>
[2] 
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBLSThXSg$>

—
Alexey

On 11 Nov 2022, at 12:34, Sankar Subramaniam 
<sankar.subramaniam01....@bbc.co.uk> wrote:

Hello there,

Good morning.

We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
working fine for few data pipelines.

Now we have got a new requirement that AWS KDA (running an application 
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
Kinesis Stream in different account.

I have followed AWS 
Documentation<https://urldefense.com/v3/__https:/docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBEETAzMw$>
 to grant required permission and policies for the AWS KDA and in Apache Bean 
implementation we have something like below,

KinesisIO.read()
    .withStreamName(getInputPattern())
    .withAWSClientsProvider(
        new KinesisClientsProvider(
            Regions.fromName(getAwsRegion()),
            getAwsCredentialsProvider(),
            getAwsVerifyCertificate(),
            getAwsKinesisServiceEndpoint(),
            getAwsCloudwatchServiceEndpoint()))

Here, we could only set the Kinesis Stream and not the ARN. With the above 
implementation, this application couldn’t read from the stream and from the 
logs we are seeing it’s trying to connect to the stream in the same AWS 
Account. The ARN formed using streamName assumes it’s in the same AWS Account 
whereas we want to connect to Kinesis Stream in another AWS Account.

Note: We are using ‘DefaultAWSCredentialsProviderChain’.

With this situation, wondering am I missing something / doing incorrectly here. 
Could you please give us some pointers how to use Beam to read from a 
(Kinesis)Stream in different AWS Account. Thanks.

Regards,
Sankar


As a recipient of an email from Talend, your contact personal data will be on 
our systems. Please see our privacy notice. <https://www.talend.com/privacy/>




----------------------------

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal 
views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on 
it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

Reply via email to