Re: Reading from AWS Kinesis Stream Cross account

2022-11-15 Thread Sankar Subramaniam 
Thanks Alexey and Moritz for your valuable inputs.

We were using ‘STSAssumeRoleSessionCredentialsProvider’ because of AWS SDKv1. 
Now as per response and the issue, we will upgrade AWS SDK and Beam SDK and 
give it a go. Will keep you posted on this in few days time. Thanks.

Regards,
Sankar

From: Moritz Mack 
Reply to: "user@beam.apache.org" 
Date: Tuesday, 15 November 2022 at 10:45
To: Alexey Romanenko , "user@beam.apache.org" 

Subject: Re: Reading from AWS Kinesis Stream Cross account

Hi Sankar,

First, as Alexey pointed out, please try and migrate to the Beam AWS SDK v2 as 
soon as possible. The SDK v1 (including the Kinesis module) has been long 
deprecated and will be removed some time soon.

The AWS API doesn’t support cross-account access for Kinesis using an ARN. This 
is always based on the stream name as shown in the example you’ve linked [1].  
You must use STS / assume role credentials to do this, it can’t be done using 
DefaultAWSCredentialsProviderChain.

Assuming you’ve correctly configured all required policies and roles (on both 
accounts) following [1], you can then use the StsAssumeRoleCredentialsProvider. 
For the Beam AWS SDK v2 this can be done using pipeline options [2], or 
programmatically of course.
Support for StsAssumeRoleCredentialsProvider is more limited in SDK v1, though 
it should also work.

--awsCredentialsProvider={
   "@type": "StsAssumeRoleCredentialsProvider",
   "roleArn": "arn:aws:iam::SOURCE01234567:role/KA-Source-Stream-Role",
   "roleSessionName": "ksassumedrolesession"
}

The STS client will implicitly use the DefaultAWSCredentialsProviderChain to 
assume the above role of the source account using an authenticated principal of 
the sink account (source/sink nomenclature as used in [1]). Please make sure 
your environment is configured accordingly, this part is easy to miss.

Let me know if you have more questions!
Regards,
Moritz


[1] https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html
[2] 
https://beam.apache.org/releases/javadoc/2.42.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html#getAwsCredentialsProvider--

On 14.11.22, 19:05, "Alexey Romanenko"  wrote:

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 
[1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since 
former is

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO.

PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s 
highly recommended to switch to one based on AWS SDK v2 [2] since former is 
deprecated.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTCweGmqKQ$>
[2] 
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBLSThXSg$>

—
Alexey

On 11 Nov 2022, at 12:34, Sankar Subramaniam 
 wrote:

Hello there,

Good morning.

We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
working fine for few data pipelines.

Now we have got a new requirement that AWS KDA (running an application 
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
Kinesis Stream in different account.

I have followed AWS 
Documentation<https://urldefense.com/v3/__https:/docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBEETAzMw$>
 to grant required permission and policies for the AWS KDA and in Apache Bean 
implementation we have something like below,

KinesisIO.read()
.withStreamName(getInputPattern())
.withAWSClientsProvider(
new KinesisClientsProvider(
Regions.fromName(getAwsRegion()),
getAwsCredentialsProvider(),
getAwsVerifyCertificate(),
getAwsKinesisServiceEndpoint(),
getAwsCloudwatchServiceEndpoint()))

Here, we could only set the Kinesis Stream and not the ARN. With the above 
implementation, this application couldn’t read from the stream and from the 
logs we are seeing it’s trying to connect to the stream in the same AWS 
Account.

Re: Reading from AWS Kinesis Stream Cross account

2022-11-15 Thread Moritz Mack 
Hi Sankar,

First, as Alexey pointed out, please try and migrate to the Beam AWS SDK v2 as 
soon as possible. The SDK v1 (including the Kinesis module) has been long 
deprecated and will be removed some time soon.

The AWS API doesn’t support cross-account access for Kinesis using an ARN. This 
is always based on the stream name as shown in the example you’ve linked [1].  
You must use STS / assume role credentials to do this, it can’t be done using 
DefaultAWSCredentialsProviderChain.

Assuming you’ve correctly configured all required policies and roles (on both 
accounts) following [1], you can then use the StsAssumeRoleCredentialsProvider. 
For the Beam AWS SDK v2 this can be done using pipeline options [2], or 
programmatically of course.
Support for StsAssumeRoleCredentialsProvider is more limited in SDK v1, though 
it should also work.

--awsCredentialsProvider={
   "@type": "StsAssumeRoleCredentialsProvider",
   "roleArn": "arn:aws:iam::SOURCE01234567:role/KA-Source-Stream-Role",
   "roleSessionName": "ksassumedrolesession"
}

The STS client will implicitly use the DefaultAWSCredentialsProviderChain to 
assume the above role of the source account using an authenticated principal of 
the sink account (source/sink nomenclature as used in [1]). Please make sure 
your environment is configured accordingly, this part is easy to miss.

Let me know if you have more questions!
Regards,
Moritz


[1] https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html
[2] 
https://beam.apache.org/releases/javadoc/2.42.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html#getAwsCredentialsProvider--

On 14.11.22, 19:05, "Alexey Romanenko"  wrote:

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 
[1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since 
former is

If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO.

PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s 
highly recommended to switch to one based on AWS SDK v2 [2] since former is 
deprecated.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
[2] 
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java

—
Alexey


On 11 Nov 2022, at 12:34, Sankar Subramaniam 
 wrote:

Hello there,

Good morning.

We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
working fine for few data pipelines.

Now we have got a new requirement that AWS KDA (running an application 
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
Kinesis Stream in different account.

I have followed AWS 
Documentation
 to grant required permission and policies for the AWS KDA and in Apache Bean 
implementation we have something like below,

KinesisIO.read()
.withStreamName(getInputPattern())
.withAWSClientsProvider(
new KinesisClientsProvider(
Regions.fromName(getAwsRegion()),
getAwsCredentialsProvider(),
getAwsVerifyCertificate(),
getAwsKinesisServiceEndpoint(),
getAwsCloudwatchServiceEndpoint()))

Here, we could only set the Kinesis Stream and not the ARN. With the above 
implementation, this application couldn’t read from the stream and from the 
logs we are seeing it’s trying to connect to the stream in the same AWS 
Account. The ARN formed using streamName assumes it’s in the same AWS Account 
whereas we want to connect to Kinesis Stream in another AWS Account.

Note: We are using ‘DefaultAWSCredentialsProviderChain’.

With this situation, wondering am I missing something / doing incorrectly here. 
Could you please give us some pointers how to use Beam to read from a 
(Kinesis)Stream in different AWS Account. Thanks.

Regards,
Sankar


As a recipient of an email from Talend, your contact personal data will be on 
our systems. Please see our privacy notice.

Re: Reading from AWS Kinesis Stream Cross account

2022-11-14 Thread Alexey Romanenko 
If I’m not mistaken, it’s not supported in the current implementation of 
KinesisIO.

PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s 
highly recommended to switch to one based on AWS SDK v2 [2] since former is 
deprecated.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
[2] 
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java

—
Alexey

> On 11 Nov 2022, at 12:34, Sankar Subramaniam 
>  wrote:
> 
> Hello there,
>  
> Good morning.
>  
> We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
> AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
> working fine for few data pipelines.
>  
> Now we have got a new requirement that AWS KDA (running an application 
> implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
> Kinesis Stream in different account.
>  
> I have followed AWS Documentation 
> 
>  to grant required permission and policies for the AWS KDA and in Apache Bean 
> implementation we have something like below,
>  
> KinesisIO.read()
> .withStreamName(getInputPattern())
> .withAWSClientsProvider(
> new KinesisClientsProvider(
> Regions.fromName(getAwsRegion()),
> getAwsCredentialsProvider(),
> getAwsVerifyCertificate(),
> getAwsKinesisServiceEndpoint(),
> getAwsCloudwatchServiceEndpoint()))
>  
> Here, we could only set the Kinesis Stream and not the ARN. With the above 
> implementation, this application couldn’t read from the stream and from the 
> logs we are seeing it’s trying to connect to the stream in the same AWS 
> Account. The ARN formed using streamName assumes it’s in the same AWS Account 
> whereas we want to connect to Kinesis Stream in another AWS Account.
>  
> Note: We are using ‘DefaultAWSCredentialsProviderChain’.
>  
> With this situation, wondering am I missing something / doing incorrectly 
> here. Could you please give us some pointers how to use Beam to read from a 
> (Kinesis)Stream in different AWS Account. Thanks.
>  
> Regards,
> Sankar

Reading from AWS Kinesis Stream Cross account

2022-11-11 Thread Sankar Subramaniam 
Hello there,

Good morning.

We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
working fine for few data pipelines.

Now we have got a new requirement that AWS KDA (running an application 
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
Kinesis Stream in different account.

I have followed AWS 
Documentation
 to grant required permission and policies for the AWS KDA and in Apache Bean 
implementation we have something like below,

KinesisIO.read()
.withStreamName(getInputPattern())
.withAWSClientsProvider(
new KinesisClientsProvider(
Regions.fromName(getAwsRegion()),
getAwsCredentialsProvider(),
getAwsVerifyCertificate(),
getAwsKinesisServiceEndpoint(),
getAwsCloudwatchServiceEndpoint()))

Here, we could only set the Kinesis Stream and not the ARN. With the above 
implementation, this application couldn’t read from the stream and from the 
logs we are seeing it’s trying to connect to the stream in the same AWS 
Account. The ARN formed using streamName assumes it’s in the same AWS Account 
whereas we want to connect to Kinesis Stream in another AWS Account.

Note: We are using ‘DefaultAWSCredentialsProviderChain’.

With this situation, wondering am I missing something / doing incorrectly here. 
Could you please give us some pointers how to use Beam to read from a 
(Kinesis)Stream in different AWS Account. Thanks.

Regards,
Sankar