RE: Rebalancing issue while Kafka scaling

2016-06-01 Thread Thakrar, Jayesh
--- From: Hafsa Asif [mailto:hafsa.a...@matchinguu.com] Sent: Wednesday, June 01, 2016 7:05 AM To: users@kafka.apache.org Cc: Spico Florin <spicoflo...@gmail.com> Subject: Re: Rebalancing issue while Kafka scaling Just for more info: If I have 10 servers in a cluster, so for the most tolerant cluste

Re: Rebalancing issue while Kafka scaling

2016-06-01 Thread Hafsa Asif
ment, > >> if > >>>> all your topics have atleast 2 insync replicas, including the server > >> being > >>>> removed and you intend to rebalance after server removal). > >>>> > >>>> However, "automating" the rebalanci

Re: Rebalancing issue while Kafka scaling

2016-06-01 Thread Ben Stopford
>>> >>>> However, "automating" the rebalancing of topic partitions is not >> trivial. >>>> >>>> There is a KIP out there to help with the rebalancing , but lacks >> details >>>> - >>>> >> https://c

Re: Rebalancing issue while Kafka scaling

2016-06-01 Thread Hafsa Asif
but lacks > details > >> - > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+rebalancing > >> My guess is due to its non-trivial nature AND the number of cases one > >> needs to take care of - e.g. scaling up by

Re: Rebalancing issue while Kafka scaling

2016-06-01 Thread Ben Stopford
reness, the task becomes even more >> involved. >> >> Regards, >> Jayesh >> >> -Original Message- >> From: Spico Florin [mailto:spicoflo...@gmail.com] >> Sent: Tuesday, May 31, 2016 9:44 AM >> To: users@kafka.apache.org >>

Re: Rebalancing issue while Kafka scaling

2016-06-01 Thread Hafsa Asif
de cluster. > Furthermore, to be really effective, one needs to be cognizant of the > partition sizes, and with rack-awareness, the task becomes even more > involved. > > Regards, > Jayesh > > -Original Message- > From: Spico Florin [mailto:spicoflo...@gmail.com] > Sen

RE: Rebalancing issue while Kafka scaling

2016-05-31 Thread Thakrar, Jayesh
- From: Spico Florin [mailto:spicoflo...@gmail.com] Sent: Tuesday, May 31, 2016 9:44 AM To: users@kafka.apache.org Subject: Re: Rebalancing issue while Kafka scaling Hi! What version of Kafka you are using? What do you mean by "Kafka needs rebalacing?" Rebalancing of what? Can you ple

Re: Rebalancing issue while Kafka scaling

2016-05-31 Thread Spico Florin
Hi! What version of Kafka you are using? What do you mean by "Kafka needs rebalacing?" Rebalancing of what? Can you please be more specific. Regards, Florin On Tue, May 31, 2016 at 4:58 PM, Hafsa Asif wrote: > Hello Folks, > > Today , my team members shows

Rebalancing issue while Kafka scaling

2016-05-31 Thread Hafsa Asif
Hello Folks, Today , my team members shows concern that whenever we increase node in Kafka cluster, Kafka needs rebalancing. The rebalancing is sort of manual and not-good step whenever scaling happens. Second, if Kafka scales up then it cannot be scale down. Please provide us proper guidance