The consumers in consumer group 'X' do not have a regex subscription matching the newly created topic 'C'. They simply subscribe with the subscribe(java.util.Collection<java.lang.String> topics) method on topics 'A' and 'B'.
Shouldn't the consumer group have a different state from "Stable" during a rebalancing regardless of the cause? How else can we determine the consumer lag of the group during the rebalancing? Best regards, Wouter Have a look at our brand NEW job website: jobs.aca-it.be ! *ACA IT-Solutions NV* *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt T +32(0)11 26 50 10 | F +32(0)11 26 50 11 www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> | Linkedin <http://www.linkedin.com/company/6524> On 5 February 2018 at 00:13, Hans Jespersen <h...@confluent.io> wrote: > Do the consumers in consumer group ‘X’ have a regex subscription that > matches the newly created topic ‘C’? > > If they do then they will only discover this new topic once their ‘ > metadata.max.age.ms’ metadata refresh interval has passed, which > defaults to 5 minutes. > > metadata.max.age.ms The period of time in milliseconds after which we > force a refresh of metadata even if we haven't seen any partition > leadership changes to proactively discover any new brokers or partitions > -hans > > > > On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be> > wrote: > > > > Hi Hans, > > > > Thanks for the response! > > > > However, I get this result for all topics, not just for the newly created > > topic. > > > > Situation sketch: > > 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with > > partition assignments and lag information. Consumer group 'X' is > "Stable". > > 2a. Topic 'C' is (being) created. > > 2b. During this creation, I do not have a partition assignment for > consumer > > group 'X' for topics 'A' and 'B' but the consumer group is still > "Stable". > > 3. A second later: I have a partition assignment for consumer group 'X' > for > > topics 'A' and 'B' again and the consumer group is still "Stable". > > > > I expected the state of consumer group 'X' during step 2b to be > > "PreparingRebalance" or "AwaitingSync". > > > > Best regards, > > Wouter > > > >> On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote: > >> > >> I believe this is expected behavior. > >> > >> If there are no subscriptions to a new topic, and therefor no partition > >> assignments, and definitely no committed offsets, then lag is an > undefined > >> concept. When the consumers subscribe to this new topic they may chose > to > >> start at the beginning or end of the commit log so the lag cannot be > >> predicted in advance. > >> > >> -hans > >> > >>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be> > >> wrote: > >>> > >>> Can anyone clarify if this is a bug in Kafka or the expected behavior? > >>> > >>> Best regards, > >>> Wouter > >>> > >>> > >>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.banc...@aca-it.be> > >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> I'm trying to write an external tool to monitor consumer lag on Apache > >>>> Kafka. > >>>> > >>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch > the > >>>> consumer offsets. > >>>> > >>>> When using this tool, partition assignments seem to be unavailable > >>>> temporarily during the creation of a new topic even if the consumer > >> group > >>>> has no subscription on this new topic. This seems to match the > >>>> documentation > >>>> <https://cwiki.apache.org/confluence/display/KAFKA/ > >> Kafka+Client-side+Assignment+Proposal> > >>>> saying *"Topic metadata changes which have no impact on subscriptions > >>>> cause resync"*. > >>>> > >>>> However, when this occurs I'd expect the state of the consumer to be > >>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable". > >>>> > >>>> Is this a bug in the tooling or is there a different way to obtain the > >>>> correct offsets for a consumer group during a rebalance? > >>>> > >>>> I'm using Kafka 10.2.1 but I haven't found any related issues in > recent > >>>> changelogs. > >>>> Best regards, > >>>> Wouter > >>>> > >> >
