Hi Sankar, First, as Alexey pointed out, please try and migrate to the Beam AWS SDK v2 as soon as possible. The SDK v1 (including the Kinesis module) has been long deprecated and will be removed some time soon.
The AWS API doesn’t support cross-account access for Kinesis using an ARN. This is always based on the stream name as shown in the example you’ve linked [1]. You must use STS / assume role credentials to do this, it can’t be done using DefaultAWSCredentialsProviderChain. Assuming you’ve correctly configured all required policies and roles (on both accounts) following [1], you can then use the StsAssumeRoleCredentialsProvider. For the Beam AWS SDK v2 this can be done using pipeline options [2], or programmatically of course. Support for StsAssumeRoleCredentialsProvider is more limited in SDK v1, though it should also work. --awsCredentialsProvider={ "@type": "StsAssumeRoleCredentialsProvider", "roleArn": "arn:aws:iam::SOURCE01234567:role/KA-Source-Stream-Role", "roleSessionName": "ksassumedrolesession" } The STS client will implicitly use the DefaultAWSCredentialsProviderChain to assume the above role of the source account using an authenticated principal of the sink account (source/sink nomenclature as used in [1]). Please make sure your environment is configured accordingly, this part is easy to miss. Let me know if you have more questions! Regards, Moritz [1] https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html [2] https://beam.apache.org/releases/javadoc/2.42.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html#getAwsCredentialsProvider-- On 14.11.22, 19:05, "Alexey Romanenko" <aromanenko....@gmail.com> wrote: If I’m not mistaken, it’s not supported in the current implementation of KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since former is If I’m not mistaken, it’s not supported in the current implementation of KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since former is deprecated. [1] https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTCweGmqKQ$> [2] https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBLSThXSg$> — Alexey On 11 Nov 2022, at 12:34, Sankar Subramaniam <sankar.subramaniam01....@bbc.co.uk> wrote: Hello there, Good morning. We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s working fine for few data pipelines. Now we have got a new requirement that AWS KDA (running an application implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS Kinesis Stream in different account. I have followed AWS Documentation<https://urldefense.com/v3/__https:/docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBEETAzMw$> to grant required permission and policies for the AWS KDA and in Apache Bean implementation we have something like below, KinesisIO.read() .withStreamName(getInputPattern()) .withAWSClientsProvider( new KinesisClientsProvider( Regions.fromName(getAwsRegion()), getAwsCredentialsProvider(), getAwsVerifyCertificate(), getAwsKinesisServiceEndpoint(), getAwsCloudwatchServiceEndpoint())) Here, we could only set the Kinesis Stream and not the ARN. With the above implementation, this application couldn’t read from the stream and from the logs we are seeing it’s trying to connect to the stream in the same AWS Account. The ARN formed using streamName assumes it’s in the same AWS Account whereas we want to connect to Kinesis Stream in another AWS Account. Note: We are using ‘DefaultAWSCredentialsProviderChain’. With this situation, wondering am I missing something / doing incorrectly here. Could you please give us some pointers how to use Beam to read from a (Kinesis)Stream in different AWS Account. Thanks. Regards, Sankar As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our privacy notice. <https://www.talend.com/privacy/>