[
https://issues.apache.org/jira/browse/FLINK-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563155#comment-16563155
]
ASF GitHub Bot commented on FLINK-9897:
---------------------------------------
tzulitai commented on issue #6408: [FLINK-9897][Kinesis Connector] Make
adaptive reads depend on run loop time instead of fetchintervalmillis
URL: https://github.com/apache/flink/pull/6408#issuecomment-409101468
+1, thanks for the work @glaksh100 and @tweise for the help on the reviews.
Merging this to master and 1.6 ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Further enhance adaptive reads in Kinesis Connector to depend on run loop time
> ------------------------------------------------------------------------------
>
> Key: FLINK-9897
> URL: https://issues.apache.org/jira/browse/FLINK-9897
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Affects Versions: 1.4.2, 1.5.1
> Reporter: Lakshmi Rao
> Assignee: Lakshmi Rao
> Priority: Major
> Labels: pull-request-available
>
> In FLINK-9692, we introduced the ability for the shardConsumer to adaptively
> read more records based on the current average record size to optimize the 2
> Mb/sec shard limit. The feature maximizes maxNumberOfRecordsPerFetch of 5
> reads/sec (as prescribed by Kinesis limits). In the case where applications
> take more time to process records in the run loop, they are no longer able to
> read at a frequency of 5 reads/sec (even though their fetchIntervalMillis
> maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch
> should be calculated based on the time that the run loop actually takes as
> opposed to fetchIntervalMillis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
