I asked a similar question a while ago. There doesn't appear to be a way to not triggering the rebalance. But I'm not sure why it would be taking > 1hr in your case. For us it was pretty fast.
https://www.mail-archive.com/users@kafka.apache.org/msg23925.html On Fri, Feb 10, 2017 at 4:28 AM, Krzysztof Lesniewski, Nexiot AG < krzysztof.lesniew...@nexiot.ch> wrote: > Would be great to get some input on it. > > - Krzysztof Lesniewski > > > On 06.02.2017 08:27, Praveen wrote: > >> I have a 16 broker kafka cluster. There is a topic with 32 partitions >> containing real time data and on the other side, I have 32 boxes w/ 1 >> consumer reading from these partitions. >> >> Today our deployment strategy is stop, deploy and start the processes on >> all the 32 consumers. This triggers re-balancing and takes a long period >> of >> time (> 1hr). Such a long pause isn't good for real time processing. >> >> I was thinking of rolling deploy but I think that will still cause >> re-balancing b/c we will still have consumers go down and come up. >> >> How do you deploy to consumers without triggering re-balancing (or >> triggering one that doesn't affect your SLA) when doing real time >> processing? >> >> Thanks, >> Praveen >> >> >
