It shouldn't be possible to have conflicting assignments. It kind of sounds like your consumers aren't all staying in the group. Maybe you could share a code snippet with what you're trying to do?
-Jason On Wed, Feb 24, 2016 at 3:47 AM, craig w <[email protected]> wrote: > I implemented the ConsumerRebalanceListener just to log the TopicPartitions > revoked and assigned. In a test, I had 5 consumers subscribed to one topic > (which had 15 partitions) all with the same group id. Upon startup they all > were assigned 3 different partitions. I pushed in 15 messages (one per > partition) and each consumer received 3 messages. All 15 messages were > processed once. > > I then pushed 90 messages (to ensure each consumer would be busy for a > bit), then I started a 6th consumer. I watched the logs of the 5 original > consumers and saw their TopicPartitions were being revoked/assigned a few > times. When the dust settled I saw duplicates subscriptions, at least > according to the ConsumerRebalanceListener. > > One consumer was assigned all 15 partitions. While other consumers had > anywhere from 2 - 4 partitions assigned. This means, multiple consumers in > the same group are consuming the same partition. Am I doing something wrong > or perhaps misunderstanding what's going on with the > ConsumberRebalanceListener? I didn't believe it was possible for consumers > in the same group subscribed to the same topic to be consuming the same > partition. > > Thanks > > On Mon, Feb 22, 2016 at 2:12 PM, craig w <[email protected]> wrote: > > > I think I'll need to do that. > > > > My current setup is where I have 200 partitions for a topic with 200 > > consumers, all part of the same group. Each consumer does a "poll". Upon > > receiving messages it will pause the topic/paritions it received, while > the > > messages are processed, as each message is processed the offset is > > committed. When all messages are processed, the topic/partition is > resumed > > and poll is called again. However, during this it's possible that the > > consumers get restarted (as part of an upgrade, etc) or a consumer dies > and > > a new one starts up. > > > > On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <[email protected]> > > wrote: > > > >> That is correct. The typical way to handle this (if you need to) is to > >> repause the partitions in the ConsumerRebalanceListener > >> onPartitionsAssigned() hook. > >> > >> -Jason > >> > >> On Mon, Feb 22, 2016 at 11:03 AM, craig w <[email protected]> wrote: > >> > >> > so if I had a bunch of consumers coming online while some consumers > had > >> > partitions paused, those consumers that were paused would become > >> unpaused? > >> > > >> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <[email protected]> > >> > wrote: > >> > > >> > > Nope. Pausing a partition just stops the consumer from sending any > >> more > >> > > fetches for it. It will not trigger a partition reassignment. One > >> thing > >> > to > >> > > be wary of, however, is that the partition will automatically be > >> unpaused > >> > > after the next normal rebalance. > >> > > > >> > > -Jason > >> > > > >> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <[email protected]> > wrote: > >> > > > >> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group > and > >> one > >> > > > consumer pauses the topic+partition it's consuming, does that > >> > allow/cause > >> > > > other consumers in the group to pick it up? > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > > >> > https://github.com/mindscratch > >> > https://www.google.com/+CraigWickesser > >> > https://twitter.com/mind_scratch > >> > https://twitter.com/craig_links > >> > > >> > > > > > > > > -- > > > > https://github.com/mindscratch > > https://www.google.com/+CraigWickesser > > https://twitter.com/mind_scratch > > https://twitter.com/craig_links > > > > > > > -- > > https://github.com/mindscratch > https://www.google.com/+CraigWickesser > https://twitter.com/mind_scratch > https://twitter.com/craig_links >
