I logged https://issues.apache.org/jira/browse/KAFKA-6557
Best regards, Wouter On 13 February 2018 at 12:50, Jan Filipiak <jan.filip...@trivago.com> wrote: > I would encourage you todo so. > I also think its not reasonable behavior > > On 13.02.2018 11:28, Wouter Bancken wrote: > >> We have upgraded our Kafka version as an attempt to solve this issue. >> However, the issue is still present in Kafka 1.0.0. >> >> Can I log a bug for this in JIRA? >> >> Wouter >> >> On 5 February 2018 at 09:22, Wouter Bancken <wouter.banc...@aca-it.be> >> wrote: >> >> The consumers in consumer group 'X' do not have a regex subscription >>> matching the newly created topic 'C'. They simply subscribe with >>> the subscribe(java.util.Collection<java.lang.String> topics) method on >>> topics 'A' and 'B'. >>> >>> Shouldn't the consumer group have a different state from "Stable" during >>> a >>> rebalancing regardless of the cause? How else can we determine the >>> consumer >>> lag of the group during the rebalancing? >>> >>> Best regards, >>> Wouter >>> >>> Have a look at our brand NEW job website: jobs.aca-it.be ! >>> >>> >>> *ACA IT-Solutions NV* >>> *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt >>> T +32(0)11 26 50 10 | F +32(0)11 26 50 11 >>> www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook >>> <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> | >>> Linkedin <http://www.linkedin.com/company/6524> >>> >>> >>> On 5 February 2018 at 00:13, Hans Jespersen <h...@confluent.io> wrote: >>> >>> Do the consumers in consumer group ‘X’ have a regex subscription that >>>> matches the newly created topic ‘C’? >>>> >>>> If they do then they will only discover this new topic once their ‘ >>>> metadata.max.age.ms’ metadata refresh interval has passed, which >>>> defaults to 5 minutes. >>>> >>>> metadata.max.age.ms The period of time in milliseconds after which >>>> we force a refresh of metadata even if we haven't seen any partition >>>> leadership changes to proactively discover any new brokers or partitions >>>> -hans >>>> >>>> >>>> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be> >>>>> >>>> wrote: >>>> >>>>> Hi Hans, >>>>> >>>>> Thanks for the response! >>>>> >>>>> However, I get this result for all topics, not just for the newly >>>>> >>>> created >>>> >>>>> topic. >>>>> >>>>> Situation sketch: >>>>> 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with >>>>> partition assignments and lag information. Consumer group 'X' is >>>>> >>>> "Stable". >>>> >>>>> 2a. Topic 'C' is (being) created. >>>>> 2b. During this creation, I do not have a partition assignment for >>>>> >>>> consumer >>>> >>>>> group 'X' for topics 'A' and 'B' but the consumer group is still >>>>> >>>> "Stable". >>>> >>>>> 3. A second later: I have a partition assignment for consumer group 'X' >>>>> >>>> for >>>> >>>>> topics 'A' and 'B' again and the consumer group is still "Stable". >>>>> >>>>> I expected the state of consumer group 'X' during step 2b to be >>>>> "PreparingRebalance" or "AwaitingSync". >>>>> >>>>> Best regards, >>>>> Wouter >>>>> >>>>> On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote: >>>>>> >>>>>> I believe this is expected behavior. >>>>>> >>>>>> If there are no subscriptions to a new topic, and therefor no >>>>>> partition >>>>>> assignments, and definitely no committed offsets, then lag is an >>>>>> >>>>> undefined >>>> >>>>> concept. When the consumers subscribe to this new topic they may chose >>>>>> >>>>> to >>>> >>>>> start at the beginning or end of the commit log so the lag cannot be >>>>>> predicted in advance. >>>>>> >>>>>> -hans >>>>>> >>>>>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be >>>>>>> >>>>>> wrote: >>>>>> >>>>>>> Can anyone clarify if this is a bug in Kafka or the expected >>>>>>> behavior? >>>>>>> >>>>>>> Best regards, >>>>>>> Wouter >>>>>>> >>>>>>> >>>>>>> On 30 January 2018 at 21:04, Wouter Bancken < >>>>>>> wouter.banc...@aca-it.be >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to write an external tool to monitor consumer lag on >>>>>>>> >>>>>>> Apache >>>> >>>>> Kafka. >>>>>>>> >>>>>>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch >>>>>>>> >>>>>>> the >>>> >>>>> consumer offsets. >>>>>>>> >>>>>>>> When using this tool, partition assignments seem to be unavailable >>>>>>>> temporarily during the creation of a new topic even if the consumer >>>>>>>> >>>>>>> group >>>>>> >>>>>>> has no subscription on this new topic. This seems to match the >>>>>>>> documentation >>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/ >>>>>>>> >>>>>>> Kafka+Client-side+Assignment+Proposal> >>>>>> >>>>>>> saying *"Topic metadata changes which have no impact on subscriptions >>>>>>>> cause resync"*. >>>>>>>> >>>>>>>> However, when this occurs I'd expect the state of the consumer to be >>>>>>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable". >>>>>>>> >>>>>>>> Is this a bug in the tooling or is there a different way to obtain >>>>>>>> >>>>>>> the >>>> >>>>> correct offsets for a consumer group during a rebalance? >>>>>>>> >>>>>>>> I'm using Kafka 10.2.1 but I haven't found any related issues in >>>>>>>> >>>>>>> recent >>>> >>>>> changelogs. >>>>>>>> Best regards, >>>>>>>> Wouter >>>>>>>> >>>>>>>> >>> >
