I implemented the ConsumerRebalanceListener just to log the TopicPartitions revoked and assigned. In a test, I had 5 consumers subscribed to one topic (which had 15 partitions) all with the same group id. Upon startup they all were assigned 3 different partitions. I pushed in 15 messages (one per partition) and each consumer received 3 messages. All 15 messages were processed once.
I then pushed 90 messages (to ensure each consumer would be busy for a bit), then I started a 6th consumer. I watched the logs of the 5 original consumers and saw their TopicPartitions were being revoked/assigned a few times. When the dust settled I saw duplicates subscriptions, at least according to the ConsumerRebalanceListener. One consumer was assigned all 15 partitions. While other consumers had anywhere from 2 - 4 partitions assigned. This means, multiple consumers in the same group are consuming the same partition. Am I doing something wrong or perhaps misunderstanding what's going on with the ConsumberRebalanceListener? I didn't believe it was possible for consumers in the same group subscribed to the same topic to be consuming the same partition. Thanks On Mon, Feb 22, 2016 at 2:12 PM, craig w <[email protected]> wrote: > I think I'll need to do that. > > My current setup is where I have 200 partitions for a topic with 200 > consumers, all part of the same group. Each consumer does a "poll". Upon > receiving messages it will pause the topic/paritions it received, while the > messages are processed, as each message is processed the offset is > committed. When all messages are processed, the topic/partition is resumed > and poll is called again. However, during this it's possible that the > consumers get restarted (as part of an upgrade, etc) or a consumer dies and > a new one starts up. > > On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <[email protected]> > wrote: > >> That is correct. The typical way to handle this (if you need to) is to >> repause the partitions in the ConsumerRebalanceListener >> onPartitionsAssigned() hook. >> >> -Jason >> >> On Mon, Feb 22, 2016 at 11:03 AM, craig w <[email protected]> wrote: >> >> > so if I had a bunch of consumers coming online while some consumers had >> > partitions paused, those consumers that were paused would become >> unpaused? >> > >> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <[email protected]> >> > wrote: >> > >> > > Nope. Pausing a partition just stops the consumer from sending any >> more >> > > fetches for it. It will not trigger a partition reassignment. One >> thing >> > to >> > > be wary of, however, is that the partition will automatically be >> unpaused >> > > after the next normal rebalance. >> > > >> > > -Jason >> > > >> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <[email protected]> wrote: >> > > >> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group and >> one >> > > > consumer pauses the topic+partition it's consuming, does that >> > allow/cause >> > > > other consumers in the group to pick it up? >> > > > >> > > >> > >> > >> > >> > -- >> > >> > https://github.com/mindscratch >> > https://www.google.com/+CraigWickesser >> > https://twitter.com/mind_scratch >> > https://twitter.com/craig_links >> > >> > > > > -- > > https://github.com/mindscratch > https://www.google.com/+CraigWickesser > https://twitter.com/mind_scratch > https://twitter.com/craig_links > > -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
