Hello,
If you are consuming from a single stream, you can use the shard ID to achieve a better distribution. Since the shard IDs are assigned incrementally like so: - shardId-000000000000 - shardId-000000000001 - shardId-000000000002 - etc You can substring the prefix and convert to a number. - shardId-000000000000 - shardId-000000000001 - shardId-000000000002 Thanks, Danny Cranmer On Mon, Jul 26, 2021 at 3:11 AM Caizhi Weng <tsreape...@gmail.com> wrote: > Hi! > > It's stated on the line just below that in the document. > > It is recommended to monitor the shard distribution and adjust assignment >> appropriately. A custom assigner implementation can be set via >> setShardAssigner(KinesisShardAssigner) to optimize the hash function or use >> static overrides to limit skew. > > > Vijayendra Yadav <contact....@gmail.com> 于2021年7月24日周六 上午9:15写道： > >> Hi Team, >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.html >> >> *" There is no perfect generic default assignment function. Default shard >> to subtask assignment, which is based on hash code, may result in skew, >> with some subtasks having many shards assigned and others none."* >> >> >> *Question*: How to reduce this SKEW ? Some of the subtasks are idle and >> others get double the load. I am trying to achieve 1 shard to 1 subtask >> match. >> >> Thanks, >> Vijay >> >
