Hi Gordon,
thanks for looking into this and sorry it took me so long to file the
issue: https://issues.apache.org/jira/browse/FLINK-6365.
Really appreciate your contributions for the Kinesis connector!
Cheers,
Steffen
On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote:
Hi Steffan,
I have to admit that I didn’t put too much thoughts in the default
values for the Kinesis consumer.
I’d say it would be reasonable to change the default values to follow
KCL’s settings. Could you file a JIRA for this?
In general, we might want to reconsider all the default values for
configs related to the getRecords call, i.e.
- SHARD_GETRECORDS_MAX
- SHARD_GETRECORDS_INTERVAL_MILLIS
- SHARD_GETRECORDS_BACKOFF_*
Cheers,
Gordon
On March 23, 2017 at 2:12:32 AM, Steffen Hausmann
(stef...@hausmann-family.de <mailto:stef...@hausmann-family.de>) wrote:
Hi there,
I recently ran into problems with a Flink job running on an EMR cluster
consuming events from a Kinesis stream receiving roughly 15k
event/second. Although the EMR cluster was substantially scaled and CPU
utilization and system load were well below any alarming threshold, the
processing of events of the stream increasingly fell behind.
Eventually, it turned out that the SHARD_GETRECORDS_MAX defaults to 100
which is apparently causing too much overhead when consuming events from
the stream. Increasing the value to 5000, a single GetRecords call to
Kinesis can retrieve up to 10k records, made the problem go away.
I wonder why the default value for SHARD_GETRECORDS_MAX is chosen so low
(100x less than it could be). The Kinesis Client Library defaults to
5000 and it's recommended to use this default value:
http://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#consumer-app-reading-slower.
Thanks for the clarification!
Cheers,
Steffen
