Reading from AWS Kinesis Stream Cross account

[Sankar Subramaniam]

Hello there,

Good morning.

We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from 
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s 
working fine for few data pipelines.

Now we have got a new requirement that AWS KDA (running an application 
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS 
Kinesis Stream in different account.

I have followed AWS 
Documentation<https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html>
 to grant required permission and policies for the AWS KDA and in Apache Bean 
implementation we have something like below,

KinesisIO.read()
    .withStreamName(getInputPattern())
    .withAWSClientsProvider(
        new KinesisClientsProvider(
            Regions.fromName(getAwsRegion()),
            getAwsCredentialsProvider(),
            getAwsVerifyCertificate(),
            getAwsKinesisServiceEndpoint(),
            getAwsCloudwatchServiceEndpoint()))

Here, we could only set the Kinesis Stream and not the ARN. With the above 
implementation, this application couldn’t read from the stream and from the 
logs we are seeing it’s trying to connect to the stream in the same AWS 
Account. The ARN formed using streamName assumes it’s in the same AWS Account 
whereas we want to connect to Kinesis Stream in another AWS Account.

Note: We are using ‘DefaultAWSCredentialsProviderChain’.

With this situation, wondering am I missing something / doing incorrectly here. 
Could you please give us some pointers how to use Beam to read from a 
(Kinesis)Stream in different AWS Account. Thanks.

Regards,
Sankar