[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[ https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204432#comment-17204432 ] Marlon Ou commented on KAFKA-10513: --- Thank you both so much for the help! I tried setting metadata.max.age.ms to a small value and it did work. The reason why I didn't see rebalance before is that the default value for metadata.max.age.ms is 5 minutes and I never waited past 5 mins for rebalancing. > Newly added topic or partitions are not assigned to running consumer groups > using static membership > --- > > Key: KAFKA-10513 > URL: https://issues.apache.org/jira/browse/KAFKA-10513 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 2.6.0 >Reporter: Marlon Ou >Priority: Major > > If consumers are polling messages from a certain topic with static membership > and we add new partitions to this topic while the consumers are running, no > partition reassignment is ever triggered (and hence messages published into > the new partitions are never consumed). > To reproduce, simply set group instance IDs on the consumers: > {code:java} > props.setProperty("group.instance.id", instanceId); > {code} > And then while the static consumers are running, use Kafka's admin client to > add more partitions to the topic: > {code:java} > adminClient.createPartitions(...) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[ https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17203622#comment-17203622 ] Guozhang Wang commented on KAFKA-10513: --- [~mou@ea] could you try setting the `metadata.max.age.ms` to a very small value and retry this scenario again? [~bchen225242] and I looked at both client and broker side code in 2.6 and we think it should still trigger rebalance. > Newly added topic or partitions are not assigned to running consumer groups > using static membership > --- > > Key: KAFKA-10513 > URL: https://issues.apache.org/jira/browse/KAFKA-10513 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 2.6.0 >Reporter: Marlon Ou >Priority: Major > > If consumers are polling messages from a certain topic with static membership > and we add new partitions to this topic while the consumers are running, no > partition reassignment is ever triggered (and hence messages published into > the new partitions are never consumed). > To reproduce, simply set group instance IDs on the consumers: > {code:java} > props.setProperty("group.instance.id", instanceId); > {code} > And then while the static consumers are running, use Kafka's admin client to > add more partitions to the topic: > {code:java} > adminClient.createPartitions(...) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[ https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17203342#comment-17203342 ] Boyang Chen commented on KAFKA-10513: - My understanding is that the consumer was supposed to check the metadata for topic partition, and rejoin as necessary when the partition number changes? > Newly added topic or partitions are not assigned to running consumer groups > using static membership > --- > > Key: KAFKA-10513 > URL: https://issues.apache.org/jira/browse/KAFKA-10513 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 2.6.0 >Reporter: Marlon Ou >Priority: Major > > If consumers are polling messages from a certain topic with static membership > and we add new partitions to this topic while the consumers are running, no > partition reassignment is ever triggered (and hence messages published into > the new partitions are never consumed). > To reproduce, simply set group instance IDs on the consumers: > {code:java} > props.setProperty("group.instance.id", instanceId); > {code} > And then while the static consumers are running, use Kafka's admin client to > add more partitions to the topic: > {code:java} > adminClient.createPartitions(...) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10513) Newly added topic or partitions are not assigned to running consumer groups using static membership
[ https://issues.apache.org/jira/browse/KAFKA-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17202991#comment-17202991 ] Guozhang Wang commented on KAFKA-10513: --- This seems to be a valid issue after reading the source code, cc [~bchen225242] to take another look. > Newly added topic or partitions are not assigned to running consumer groups > using static membership > --- > > Key: KAFKA-10513 > URL: https://issues.apache.org/jira/browse/KAFKA-10513 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 2.6.0 >Reporter: Marlon Ou >Priority: Major > > If consumers are polling messages from a certain topic with static membership > and we add new partitions to this topic while the consumers are running, no > partition reassignment is ever triggered (and hence messages published into > the new partitions are never consumed). > To reproduce, simply set group instance IDs on the consumers: > {code:java} > props.setProperty("group.instance.id", instanceId); > {code} > And then while the static consumers are running, use Kafka's admin client to > add more partitions to the topic: > {code:java} > adminClient.createPartitions(...) > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)