I think I'll need to do that. My current setup is where I have 200 partitions for a topic with 200 consumers, all part of the same group. Each consumer does a "poll". Upon receiving messages it will pause the topic/paritions it received, while the messages are processed, as each message is processed the offset is committed. When all messages are processed, the topic/partition is resumed and poll is called again. However, during this it's possible that the consumers get restarted (as part of an upgrade, etc) or a consumer dies and a new one starts up.
On Mon, Feb 22, 2016 at 2:05 PM, Jason Gustafson <[email protected]> wrote: > That is correct. The typical way to handle this (if you need to) is to > repause the partitions in the ConsumerRebalanceListener > onPartitionsAssigned() hook. > > -Jason > > On Mon, Feb 22, 2016 at 11:03 AM, craig w <[email protected]> wrote: > > > so if I had a bunch of consumers coming online while some consumers had > > partitions paused, those consumers that were paused would become > unpaused? > > > > On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson <[email protected]> > > wrote: > > > > > Nope. Pausing a partition just stops the consumer from sending any more > > > fetches for it. It will not trigger a partition reassignment. One thing > > to > > > be wary of, however, is that the partition will automatically be > unpaused > > > after the next normal rebalance. > > > > > > -Jason > > > > > > On Mon, Feb 22, 2016 at 7:19 AM, craig w <[email protected]> wrote: > > > > > > > Using kafka 0.9.0.0, if there are multiple consumers in a group and > one > > > > consumer pauses the topic+partition it's consuming, does that > > allow/cause > > > > other consumers in the group to pick it up? > > > > > > > > > > > > > > > -- > > > > https://github.com/mindscratch > > https://www.google.com/+CraigWickesser > > https://twitter.com/mind_scratch > > https://twitter.com/craig_links > > > -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
