No topic owner when using different assignment strategies
Hi team, I have two consumer instances with the same group id connecting to two different topics with 1 partition created for each. One consumer uses partition.assignment.strategy=roundrobin and the other one uses default assignment strategy. Both consumers have 1 thread spawned internally and connect kafka using createMessageStreamsByFilter. The consumer with roundrobin assignment strategy connected kafka first and had 2 topics assigned to itself and then I brought up another consumer that has default assignment strategy configured. I saw rebalancing happened in both consumers but at the end only one of the topics was assigned to the consumer that is configured roundrobin assignment strategy and no topics were assigned to the other consumer. This led to one topic missing its owner. Here is the result from zk [zk: localhost:2181(CONNECTED) 0] get /consumers/test/owners/mm-benchmark-test/0 Node does not exist: /consumers/test12345667f/owners/mm-benchmark-test/0 [zk: localhost:2181(CONNECTED) 1] get /consumers/test/owners/mm-benchmark-test1/0 test-localhost-1426605370072-904d6fba-0 The kafka version I use is 0.8.2.1 -- Regards, Tao
Re: No topic owner when using different assignment strategies
This is the corrected zk result Here is the result from zk [zk: localhost:2181(CONNECTED) 0] get /consumers/test/owners/mm-benchmark-test/0 Node does not exist: /consumers/test/owners/mm-benchmark-test/0 [zk: localhost:2181(CONNECTED) 1] get /consumers/test/owners/mm-benchmark-test1/0 test-localhost-1426605370072-904d6fba-0 On Tue, Mar 17, 2015 at 11:30 PM, tao xiao xiaotao...@gmail.com wrote: Hi team, I have two consumer instances with the same group id connecting to two different topics with 1 partition created for each. One consumer uses partition.assignment.strategy=roundrobin and the other one uses default assignment strategy. Both consumers have 1 thread spawned internally and connect kafka using createMessageStreamsByFilter. The consumer with roundrobin assignment strategy connected kafka first and had 2 topics assigned to itself and then I brought up another consumer that has default assignment strategy configured. I saw rebalancing happened in both consumers but at the end only one of the topics was assigned to the consumer that is configured roundrobin assignment strategy and no topics were assigned to the other consumer. This led to one topic missing its owner. Here is the result from zk [zk: localhost:2181(CONNECTED) 0] get /consumers/test/owners/mm-benchmark-test/0 Node does not exist: /consumers/test12345667f/owners/mm-benchmark-test/0 [zk: localhost:2181(CONNECTED) 1] get /consumers/test/owners/mm-benchmark-test1/0 test-localhost-1426605370072-904d6fba-0 The kafka version I use is 0.8.2.1 -- Regards, Tao -- Regards, Tao
Re: No topic owner when using different assignment strategies
I think this is a usability issue. It might need an extra admin tool to verify if all configuration settings are correct, even if the broker can return an error message to the consumers. Thanks, Xiao Li On Mar 17, 2015, at 5:18 PM, Jiangjie Qin j...@linkedin.com.INVALID wrote: The problem is the consumers are independent to each other. We purely depend on the same algorithm running on different consumers to achieve agreement on partition assignment. Breaking this assumption violates the design in the first place. On 3/17/15, 4:13 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Probably we should return an error response if you already have a partition assignment strategy inplace for a group and you try to use other strategy. Thanks, Mayuresh On Tue, Mar 17, 2015 at 2:10 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Yeah, using different partition assignment algorithms in the same consumer group won¹t work. Is there a particular reason you want to do this? On 3/17/15, 8:32 AM, tao xiao xiaotao...@gmail.com wrote: This is the corrected zk result Here is the result from zk [zk: localhost:2181(CONNECTED) 0] get /consumers/test/owners/mm-benchmark-test/0 Node does not exist: /consumers/test/owners/mm-benchmark-test/0 [zk: localhost:2181(CONNECTED) 1] get /consumers/test/owners/mm-benchmark-test1/0 test-localhost-1426605370072-904d6fba-0 On Tue, Mar 17, 2015 at 11:30 PM, tao xiao xiaotao...@gmail.com wrote: Hi team, I have two consumer instances with the same group id connecting to two different topics with 1 partition created for each. One consumer uses partition.assignment.strategy=roundrobin and the other one uses default assignment strategy. Both consumers have 1 thread spawned internally and connect kafka using createMessageStreamsByFilter. The consumer with roundrobin assignment strategy connected kafka first and had 2 topics assigned to itself and then I brought up another consumer that has default assignment strategy configured. I saw rebalancing happened in both consumers but at the end only one of the topics was assigned to the consumer that is configured roundrobin assignment strategy and no topics were assigned to the other consumer. This led to one topic missing its owner. Here is the result from zk [zk: localhost:2181(CONNECTED) 0] get /consumers/test/owners/mm-benchmark-test/0 Node does not exist: /consumers/test12345667f/owners/mm-benchmark-test/0 [zk: localhost:2181(CONNECTED) 1] get /consumers/test/owners/mm-benchmark-test1/0 test-localhost-1426605370072-904d6fba-0 The kafka version I use is 0.8.2.1 -- Regards, Tao -- Regards, Tao -- -Regards, Mayuresh R. Gharat (862) 250-7125
