[
https://issues.apache.org/jira/browse/FLINK-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442760#comment-16442760
]
Kailash Hassan Dayanand commented on FLINK-8944:
------------------------------------------------
[Update]: There is currently a minor issue with the listShards API in AWS which
blocks these changes [https://github.com/aws/aws-sdk-java/issues/1490] (look
at the last comment). One workaround could be get list of all shards every time
which is slightly inefficient. Waiting for sometime to allow AWS changes to go
in before submitting this.
> Use ListShards for shard discovery in the flink kinesis connector
> -----------------------------------------------------------------
>
> Key: FLINK-8944
> URL: https://issues.apache.org/jira/browse/FLINK-8944
> Project: Flink
> Issue Type: Improvement
> Reporter: Kailash Hassan Dayanand
> Priority: Minor
>
> Currently the DescribeStream AWS API used to get list of shards is has a
> restricted rate limits on AWS. (5 requests per sec per account). This is
> problematic when running multiple flink jobs all on same account since each
> subtasks calls the Describe Stream. Changing this to ListShards will provide
> more flexibility on rate limits as ListShards has a 100 requests per second
> per data stream limits.
> More details on the mailing list. https://goo.gl/mRXjKh
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
