Re: Kinesis Shards and Parallelism

2018-11-16 Thread shkob1
Actually was looking at the task manager level, i did have more slots than shards, so it does make sense i had an idle task manager while other task managers split the shards between their slots Thanks! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Kinesis Shards and Parallelism

2018-11-13 Thread Tzu-Li (Gordon) Tai
Hi, Another detail not that apparent in the description is that the assignment would only be evenly distributed assuming that the open Kinesis shards have consecutive shard ids, and are of the same Kinesis stream. Once you reshard a Kinesis stream, it could be that the shard ids are no longer

Kinesis Shards and Parallelism

2018-11-12 Thread shkob1
Looking at the doc https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kinesis.html "When the number of shards is larger than the parallelism of the consumer, then each consumer subtask can subscribe to multiple shards; otherwise if the number of shards is smaller than the