Bowen Li created FLINK-7223:
-------------------------------
Summary: Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for
Flink-kinesis-connector
Key: FLINK-7223
URL: https://issues.apache.org/jira/browse/FLINK-7223
Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Affects Versions: 1.3.0
Reporter: Bowen Li
Assignee: Bowen Li
Priority: Minor
Fix For: 1.4.0
Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in
{{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
is the default value for Flink to call Kinesis's {{describeStream()}} API.
Problem: Right now, its value is 10,000millis (10sec), which is too short. We
ran into problems that Flink-kinesis-connector's call of {{describeStream()}}
exceeds Kinesis rate limit, and broken Flink taskmanager.
According to
http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,
"This operation has a limit of 10 transactions per second per account.". What
it means is that the 10transaction/account is a limit on a single
organization's AWS account......:( We contacted AWS Support, and confirmed
this. If you have more applications (either other Flink apps or non-Flink apps)
competing aggressively with your Flink app on this API, your Flink app breaks.
I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from
10,000millis(10sec) to preferably 300,000 (5min). Or at least 60,000 (1min) if
anyone has a solid reason arguing that 5min is too long,
This is also related to https://issues.apache.org/jira/browse/FLINK-6365
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
