Yes, store info in zookeeper would work. In new consumer since the coordinator will resides on the server side, this would be easily detected. I’m not sure if it is still worth making this change on the old consumer, though. Especially this is a backward incompatible change in a sense that all the consumers need to be upgraded to make it work. That said, I think this is a good point and should be taken into consideration for the new consumer coordinator.
Jiangjie (Becket) Qin On 3/17/15, 7:05 PM, "tao xiao" <xiaotao...@gmail.com> wrote: >The intention of this test is to check how kafka would behaves if two >different assignment strategies are set in the same consumer group. In >reality this would happen as we never know what configurations downstream >consumers would use. > >What about we store the assignment strategy in zk and send out a warning >to >consumer when doing rebalancing? but at least I'd suggest we document >somewhere to warn people not to use different assignment strategies for >the >same consumer group > >On Wed, Mar 18, 2015 at 8:28 AM, Xiao <lixiao1...@gmail.com> wrote: > >> I think this is a usability issue. It might need an extra admin tool to >> verify if all configuration settings are correct, even if the broker can >> return an error message to the consumers. >> >> Thanks, >> >> Xiao Li >> >> On Mar 17, 2015, at 5:18 PM, Jiangjie Qin <j...@linkedin.com.INVALID> >> wrote: >> >> > The problem is the consumers are independent to each other. We purely >> > depend on the same algorithm running on different consumers to achieve >> > agreement on partition assignment. Breaking this assumption violates >>the >> > design in the first place. >> > >> > On 3/17/15, 4:13 PM, "Mayuresh Gharat" <gharatmayures...@gmail.com> >> wrote: >> > >> >> Probably we should return an error response if you already have a >> >> partition >> >> assignment strategy inplace for a group and you try to use other >> strategy. >> >> >> >> Thanks, >> >> >> >> Mayuresh >> >> >> >> On Tue, Mar 17, 2015 at 2:10 PM, Jiangjie Qin >><j...@linkedin.com.invalid >> > >> >> wrote: >> >> >> >>> Yeah, using different partition assignment algorithms in the same >> >>> consumer >> >>> group won¹t work. Is there a particular reason you want to do this? >> >>> >> >>> On 3/17/15, 8:32 AM, "tao xiao" <xiaotao...@gmail.com> wrote: >> >>> >> >>>> This is the corrected zk result >> >>>> >> >>>> Here is the result from zk >> >>>> [zk: localhost:2181(CONNECTED) 0] get >> >>>> /consumers/test/owners/mm-benchmark-test/0 >> >>>> >> >>>> Node does not exist: /consumers/test/owners/mm-benchmark-test/0 >> >>>> >> >>>> [zk: localhost:2181(CONNECTED) 1] get >> >>>> /consumers/test/owners/mm-benchmark-test1/0 >> >>>> >> >>>> test-localhost-1426605370072-904d6fba-0 >> >>>> >> >>>> On Tue, Mar 17, 2015 at 11:30 PM, tao xiao <xiaotao...@gmail.com> >> >>> wrote: >> >>>> >> >>>>> Hi team, >> >>>>> >> >>>>> I have two consumer instances with the same group id connecting to >> >>> two >> >>>>> different topics with 1 partition created for each. One consumer >>uses >> >>>>> partition.assignment.strategy=roundrobin and the other one uses >> >>> default >> >>>>> assignment strategy. Both consumers have 1 thread spawned >>internally >> >>> and >> >>>>> connect kafka using createMessageStreamsByFilter. >> >>>>> The consumer with roundrobin assignment strategy connected kafka >> >>> first >> >>>>> and >> >>>>> had 2 topics assigned to itself and then I brought up another >> >>> consumer >> >>>>> that >> >>>>> has default assignment strategy configured. I saw rebalancing >> >>> happened >> >>>>> in >> >>>>> both consumers but at the end only one of the topics was assigned >>to >> >>> the >> >>>>> consumer that is configured roundrobin assignment strategy and no >> >>> topics >> >>>>> were assigned to the other consumer. This led to one topic missing >> >>> its >> >>>>> owner. >> >>>>> >> >>>>> Here is the result from zk >> >>>>> [zk: localhost:2181(CONNECTED) 0] get >> >>>>> /consumers/test/owners/mm-benchmark-test/0 >> >>>>> >> >>>>> Node does not exist: >> >>>>> /consumers/test12345667fffff/owners/mm-benchmark-test/0 >> >>>>> >> >>>>> [zk: localhost:2181(CONNECTED) 1] get >> >>>>> /consumers/test/owners/mm-benchmark-test1/0 >> >>>>> >> >>>>> test-localhost-1426605370072-904d6fba-0 >> >>>>> >> >>>>> The kafka version I use is 0.8.2.1 >> >>>>> >> >>>>> -- >> >>>>> Regards, >> >>>>> Tao >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Regards, >> >>>> Tao >> >>> >> >>> >> >> >> >> >> >> -- >> >> -Regards, >> >> Mayuresh R. Gharat >> >> (862) 250-7125 >> > >> >> > > >-- >Regards, >Tao
