[ https://issues.apache.org/jira/browse/FLINK-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554820#comment-16554820 ]
ASF GitHub Bot commented on FLINK-9897: --------------------------------------- GitHub user glaksh100 opened a pull request: https://github.com/apache/flink/pull/6409 Flink 9899.kinesis connector metrics ## What is the purpose of the change The purpose of this change is to add metrics to the `ShardConsumer` to get more observability into the performance of the Kinesis connector, including the enhancements introduced in [FLINK-9897](https://issues.apache.org/jira/browse/FLINK-9899) . **Important** - https://github.com/apache/flink/pull/6408 has to be merged **before** taking out this change. ## Brief change log All metrics are added as gauges. The following per-shard metrics are added. : - sleepTimeMillis - maxNumberOfRecordsPerFetch - numberOfAggregatedRecordsPerFetch - numberOfDeaggregatedRecordsPerFetch - bytesRequestedPerFetch - averageRecordSizeBytes - runLoopTimeNanos - loopFrequencyHz ## Verifying this change This change is already covered by existing tests, such as: `ShardConsumerTest`, `KinesisDataFetcherTest`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lyft/flink FLINK-9899.KinesisConnectorMetrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6409 ---- commit f333781a7c4f1a10b6120a962ff211e023bafaab Author: Lakshmi Gururaja Rao <glaksh100@...> Date: 2018-07-24T18:44:08Z [FLINK-9897] Make adaptive reads depend on run loop time instead of fetch interval millis Remove unused method commit f51703177df9afcdba3778909b1e9d8b7fa4bf46 Author: Lakshmi Gururaja Rao <glaksh100@...> Date: 2018-07-24T18:44:08Z [FLINK-9897] Make adaptive reads depend on run loop time instead of fetch interval millis commit d493097d09c6223383282ed90648853715b197ce Author: Lakshmi Gururaja Rao <glaksh100@...> Date: 2018-07-24T21:13:53Z [FLINK-9899] Add more ShardConsumer metrics Checkstyle fix ---- > Further enhance adaptive reads in Kinesis Connector to depend on run loop time > ------------------------------------------------------------------------------ > > Key: FLINK-9897 > URL: https://issues.apache.org/jira/browse/FLINK-9897 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector > Affects Versions: 1.4.2, 1.5.1 > Reporter: Lakshmi Rao > Assignee: Lakshmi Rao > Priority: Major > Labels: pull-request-available > > In FLINK-9692, we introduced the ability for the shardConsumer to adaptively > read more records based on the current average record size to optimize the 2 > Mb/sec shard limit. The feature maximizes maxNumberOfRecordsPerFetch of 5 > reads/sec (as prescribed by Kinesis limits). In the case where applications > take more time to process records in the run loop, they are no longer able to > read at a frequency of 5 reads/sec (even though their fetchIntervalMillis > maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch > should be calculated based on the time that the run loop actually takes as > opposed to fetchIntervalMillis. -- This message was sent by Atlassian JIRA (v7.6.3#76005)