Hello!
>>Which version of Kafka are you using?
1. I am using 0.8.1
>>How many partitions do you have in that topic?
2. In docs I found that number of partition must be equal or greater than
number of consumers ... so I try 2 case:
- 10k partition per broker
- 20k partition per broker
And one more question - how to force rebalance partitions to consumers in
borker (and how much it cost...)
I found in doc that consumer can "Force itself to rebalance within in its
consumer group" but can not find API call...So for now I just reconnect....
With best regards, Aleksey Ryabkov
-----Исходное сообщение-----
От: Jun Rao [mailto:[email protected]]
Отправлено: Tuesday, May 27, 2014 7:29 AM
Кому: [email protected]
Тема: Re: multi-broker topic rebalance
Which version of Kafka are you using? How many partitions do you have in that
topic?
Thanks,
Jun
On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич <
[email protected]> wrote:
>
> Hello!
>
> How can I rebalance topic within multiple brokers?
> For example:
>
> 1. I have about 30k consumers...
>
> 2. I need to use 3 broker (10k consumer per broker)
>
> 3. I use multiple (around 10) producers with default balance strategy
> Sometimes one broker have empty topic (consumer finished all message
> on broker)...But another broker have a lot of messages in topic...
>
> How to resolve such situation:
> -i cant just switch my consumers to another broker ( broker can not
> handle more than 10k consumers....) -I can not find how to rebalance
> topic within multiple brokers...
> - I can not find how to setup limit for message per topic (or
> partition) ... If such things exists I can handle such case when add
> new message to broker in consumer side...and switch to another ...
>
> Thanks in advance, Aleksey Ryabkov
>
