This appears to be a coding accident, I accidentally had started two consumers for each instance of the app I was testing. Removing that seems to have fixed what I was seeing.
On Wed, Feb 24, 2016 at 12:54 PM, Jason Gustafson <[email protected]> wrote: > It shouldn't be possible to have conflicting assignments. It kind of sounds > like your consumers aren't all staying in the group. Maybe you could share > a code snippet with what you're trying to do? > > -Jason > > On Wed, Feb 24, 2016 at 3:47 AM, craig w <[email protected]> wrote: > > > I implemented the ConsumerRebalanceListener just to log the > TopicPartitions > > revoked and assigned. In a test, I had 5 consumers subscribed to one > topic > > (which had 15 partitions) all with the same group id. Upon startup they > all > > were assigned 3 different partitions. I pushed in 15 messages (one per > > partition) and each consumer received 3 messages. All 15 messages were > > processed once. > > > > I then pushed 90 messages (to ensure each consumer would be busy for a > > bit), then I started a 6th consumer. I watched the logs of the 5 original > > consumers and saw their TopicPartitions were being revoked/assigned a few > > times. When the dust settled I saw duplicates subscriptions, at least > > according to the ConsumerRebalanceListener. > > > > One consumer was assigned all 15 partitions. While other consumers had > > anywhere from 2 - 4 partitions assigned. This means, multiple consumers > in > > the same group are consuming the same partition. Am I doing something > wrong > > or perhaps misunderstanding what's going on with the > > ConsumberRebalanceListener? I didn't believe it was possible for > consumers > > in the same group subscribed to the same topic to be consuming the same > > partition. > > > > Thanks > > > > On Mon, Feb 22, 2016 at 2:12 PM, craig w <[email protected]> wrote: > > > > > I think I'll need to do that. > > > > > > My current setup is where I have 200 partitions for a topic with 200 > > > consumers, all part of the same group. Each consumer does a "poll". > Upon > > > receiving messages it will pause the topic/paritions it received, while > > the > > > messages are processed, as each message is processed the offset is > > > committed. When all messages are processed, the topic/partition is > > resumed > > > and poll is called again. However, during this it's possible that the > > > consumers get restarted (as part of an upgrade, etc) or a consumer dies > > and > > > a new one starts up. > > > > > > On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <[email protected]> > > > wrote: > > > > > >> That is correct. The typical way to handle this (if you need to) is to > > >> repause the partitions in the ConsumerRebalanceListener > > >> onPartitionsAssigned() hook. > > >> > > >> -Jason > > >> > > >> On Mon, Feb 22, 2016 at 11:03 AM, craig w <[email protected]> > wrote: > > >> > > >> > so if I had a bunch of consumers coming online while some consumers > > had > > >> > partitions paused, those consumers that were paused would become > > >> unpaused? > > >> > > > >> > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson < > [email protected]> > > >> > wrote: > > >> > > > >> > > Nope. Pausing a partition just stops the consumer from sending any > > >> more > > >> > > fetches for it. It will not trigger a partition reassignment. One > > >> thing > > >> > to > > >> > > be wary of, however, is that the partition will automatically be > > >> unpaused > > >> > > after the next normal rebalance. > > >> > > > > >> > > -Jason > > >> > > > > >> > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <[email protected]> > > wrote: > > >> > > > > >> > > > Using kafka 0.9.0.0, if there are multiple consumers in a group > > and > > >> one > > >> > > > consumer pauses the topic+partition it's consuming, does that > > >> > allow/cause > > >> > > > other consumers in the group to pick it up? > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > > > >> > https://github.com/mindscratch > > >> > https://www.google.com/+CraigWickesser > > >> > https://twitter.com/mind_scratch > > >> > https://twitter.com/craig_links > > >> > > > >> > > > > > > > > > > > > -- > > > > > > https://github.com/mindscratch > > > https://www.google.com/+CraigWickesser > > > https://twitter.com/mind_scratch > > > https://twitter.com/craig_links > > > > > > > > > > > > -- > > > > https://github.com/mindscratch > > https://www.google.com/+CraigWickesser > > https://twitter.com/mind_scratch > > https://twitter.com/craig_links > > > -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
