Re: Horizontally Scaling Kafka Consumers

2015-04-30 Thread Sharninder
another process will decide when to spawn new customers. -- Sharninder On Wed, Apr 29, 2015 at 11:58 PM, Nimi Wariboko Jr wrote: > Hi, > > I was wondering what options there are/what other people are doing for > horizontally scaling kafka consumers? Basically if I have 100 partiti

Re: Horizontally Scaling Kafka Consumers

2015-04-30 Thread Evan Huus
(its connection to ZK is dropped), > > consumers > > > > from the group will be rebalanced so one passively waiting consumer > > will > > > > become active. > > > > > > > > Kind regards, > > > > Stevo Slavic. > > >

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread Joe Stein
e active > consumer > > > > from same group goes offline (its connection to ZK is dropped), > > consumers > > > > from the group will be rebalanced so one passively waiting consumer > > will > > > > become active. > > > > > > &g

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread Nimi Wariboko Jr
ill > > > become active. > > > > > > Kind regards, > > > Stevo Slavic. > > > > > > On Wed, Apr 29, 2015 at 2:25 PM, David Corley > > > wrote: > > > > > > > If the 100 partitions are all for the same topic, you can ha

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread Joe Stein
or the same topic, you can have up to > 100 > > > consumers working as part of a single consumer group for that topic. > > > You cannot have more consumers than there are partitions within a given > > > consumer group. > > > > > > On 29 April 2015

Horizontally Scaling Kafka Consumers

2015-04-29 Thread Nimi Wariboko Jr
Hi, I was wondering what options there are/what other people are doing for horizontally scaling kafka consumers? Basically if I have 100 partitions and 10 consumers, and want to temporarily scale up to 50 consumers, what can I do? So far I've thought of just simply tracking consumer membe

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread David Corley
gt; > You cannot have more consumers than there are partitions within a given > > consumer group. > > > > On 29 April 2015 at 08:41, Nimi Wariboko Jr > wrote: > > > > > Hi, > > > > > > I was wondering what options there are for horizontally

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread Stevo Slavić
as wondering what options there are for horizontally scaling kafka > > consumers? Basically if I have 100 partitions and 10 consumers, and want > to > > temporarily scale up to 50 consumers, what options do I have? > > > > So far I've thought of just simply tracking consume

Re: Horizontally Scaling Kafka Consumers

2015-04-29 Thread David Corley
t; > I was wondering what options there are for horizontally scaling kafka > consumers? Basically if I have 100 partitions and 10 consumers, and want to > temporarily scale up to 50 consumers, what options do I have? > > So far I've thought of just simply tracking consumer me

Horizontally Scaling Kafka Consumers

2015-04-29 Thread Nimi Wariboko Jr
Hi, I was wondering what options there are for horizontally scaling kafka consumers? Basically if I have 100 partitions and 10 consumers, and want to temporarily scale up to 50 consumers, what options do I have? So far I've thought of just simply tracking consumer membership somehow (e