Re: Threading Model for Kinesis

Perfect - This explains it very clearly. Thank you very much! Sameer On Tue, Aug 23, 2016 at 9:31 AM, Tzu-Li (Gordon) Tai wrote: > Slight misunderstanding here. The one thread per Kafka broker happens > *after* the assignment of Kafka partitions to the source instances. So,

Re: Threading Model for Kinesis

Hi Sameer, I realized you might be a bit confused between “source instances (which in general are Flink tasks)” and “threads” in my previous explanations. The per-broker threads in the Kafka consumer and per-shard threads in the Kinesis consumer I mentioned are threads created by the source

Re: Threading Model for Kinesis

Slight misunderstanding here. The one thread per Kafka broker happens *after* the assignment of Kafka partitions to the source instances. So, with a total of 10 partitions and 10 source instances, each source instance will first be assigned 1 partition. Then, each source instance will create 1

Re: Threading Model for Kinesis

Gordon, I tried the following with Kafka - 1 Broker but a topic has 10 partitions. I have a parallelism of 10 defined for the job. I see all my 10 source->Mapper->assignTimestamps receiving and sending data. If there is only one source instance per broker how does that happen? Thanks, Sameer On

Re: Threading Model for Kinesis

Thanks Gordon - Appreciate the fast response. Sameer On Tue, Aug 23, 2016 at 7:17 AM, Tzu-Li (Gordon) Tai wrote: > Hi! > > Kinesis shards should be ideally evenly assigned to the source instances. > So, with your example of source parallelism of 10 and 20 shards, each >

Re: Threading Model for Kinesis

Hi! Kinesis shards should be ideally evenly assigned to the source instances. So, with your example of source parallelism of 10 and 20 shards, each source instance will have 2 shards and will have 2 threads consuming them (therefore, not in round robin). For the Kafka consumer, in the source

Threading Model for Kinesis

Hi, The documentation says that there will be one thread per shard. If I my streaming job runs with a parallelism of 10 and there are 20 shards, are more threads going to be launched within a task slot running a source function to consume the additional shards or will one source function