Do the consumers in consumer group ‘X’ have a regex subscription that matches the newly created topic ‘C’?
If they do then they will only discover this new topic once their ‘metadata.max.age.ms’ metadata refresh interval has passed, which defaults to 5 minutes. metadata.max.age.ms The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions -hans > On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be> wrote: > > Hi Hans, > > Thanks for the response! > > However, I get this result for all topics, not just for the newly created > topic. > > Situation sketch: > 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with > partition assignments and lag information. Consumer group 'X' is "Stable". > 2a. Topic 'C' is (being) created. > 2b. During this creation, I do not have a partition assignment for consumer > group 'X' for topics 'A' and 'B' but the consumer group is still "Stable". > 3. A second later: I have a partition assignment for consumer group 'X' for > topics 'A' and 'B' again and the consumer group is still "Stable". > > I expected the state of consumer group 'X' during step 2b to be > "PreparingRebalance" or "AwaitingSync". > > Best regards, > Wouter > >> On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote: >> >> I believe this is expected behavior. >> >> If there are no subscriptions to a new topic, and therefor no partition >> assignments, and definitely no committed offsets, then lag is an undefined >> concept. When the consumers subscribe to this new topic they may chose to >> start at the beginning or end of the commit log so the lag cannot be >> predicted in advance. >> >> -hans >> >>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be> >> wrote: >>> >>> Can anyone clarify if this is a bug in Kafka or the expected behavior? >>> >>> Best regards, >>> Wouter >>> >>> >>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.banc...@aca-it.be> >>> wrote: >>> >>>> Hi, >>>> >>>> I'm trying to write an external tool to monitor consumer lag on Apache >>>> Kafka. >>>> >>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch the >>>> consumer offsets. >>>> >>>> When using this tool, partition assignments seem to be unavailable >>>> temporarily during the creation of a new topic even if the consumer >> group >>>> has no subscription on this new topic. This seems to match the >>>> documentation >>>> <https://cwiki.apache.org/confluence/display/KAFKA/ >> Kafka+Client-side+Assignment+Proposal> >>>> saying *"Topic metadata changes which have no impact on subscriptions >>>> cause resync"*. >>>> >>>> However, when this occurs I'd expect the state of the consumer to be >>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable". >>>> >>>> Is this a bug in the tooling or is there a different way to obtain the >>>> correct offsets for a consumer group during a rebalance? >>>> >>>> I'm using Kafka 10.2.1 but I haven't found any related issues in recent >>>> changelogs. >>>> Best regards, >>>> Wouter >>>> >>
