[
https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319310#comment-16319310
]
ASF GitHub Bot commented on KAFKA-6287:
---------------------------------------
hachikuji opened a new pull request #4407: KAFKA-6287: Consumer group command
should list simple consumer groups
URL: https://github.com/apache/kafka/pull/4407
With this patch, simple consumer groups which use Kafka for offset storage
will be viewable using the `--list` option. In addition, this patch fixes a bug
in the offset loading logic which caused us to lose the protocol type of empty
groups on coordinator failover. I also did some cleanup of the various consumer
group command test cases.
For testing, I have added new integration tests which cover listing and
describing simple consumer groups. I also added unit tests to cover loading
empty groups with assertions on the protocol type.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent protocol type for empty consumer groups
> ----------------------------------------------------
>
> Key: KAFKA-6287
> URL: https://issues.apache.org/jira/browse/KAFKA-6287
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 1.0.0
> Reporter: Ryan Leslie
> Assignee: Jason Gustafson
> Priority: Minor
>
> When a consumer is created for a new group, the group metadata's protocol
> type is set to 'consumer' and this is stored both in __consumer_offsets as
> well as in the coordinator's local cache.
> If the consumer leaves the group and the group becomes empty, ListGroups
> requests will continue to show the group as type 'consumer', and as such
> kafka-consumer-groups.sh will show it via --list.
> However, if the coordinator (broker) node is killed and a new coordinator is
> elected, when the GroupMetadataManager loads the group from
> __consumer_offsets into its cache, it will not set the protocolType if there
> are no active consumers. As a result, the group's protocolType will now
> become the empty string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh
> will no longer show the group.
> Ideally bouncing a broker should not result in the group's protocol changing.
> protocolType can be set in GroupMetadataManager.readGroupMessageValue() to
> always reflect what's present in the persistent metadata (__consumer_offsets)
> regardless if there are active members.
> In general, things can get confusing when distinguishing between 'consumer'
> and non-consumer groups. For example, DescribeGroups and OffsetFetchRequest
> does not filter on protocol type, so it's possible for
> kafka-consumer-groups.sh to describe groups (--describe) without actually
> being able to list them. In the protocol guide, OffsetFetchRequest /
> OffsetCommitRequest have their parameters listed as 'ConsumerGroup', but in
> reality these can be used for groups of unknown type as well. For instance, a
> consumer group can be copied by finding a coordinator
> (GroupCoordinatorRequest / FindCoordinatorRequest) and sending an
> OffsetCommitRequest. The group will be auto-created with an empty protocol.
> Although this is arguably correct, the group will now exist but not be a
> proper 'consumer' group until later when there is a JoinGroupRequest. Again,
> this can be confusing as far as categorization / visibility of the group is
> concerned. A group can instead be copied more directly by creating a consumer
> and calling commitSync (as kafka-consumer-groups.sh), but this does involve
> extra connections / requests and so is a little slower when trying to keep a
> large number of groups in sync in real-time across clusters.
> If we want to make it easier to keep consumer groups consistent, options
> include allowing groups to be explicitly created with a protocol type instead
> of only lazily, or for groups created outside of JoinGroupRequest the
> coordinator can gain a broker config to set a default protocol type for
> groups. This would default to 'consumer'. This sort of setting might be
> confusing for users though, since implicitly created groups is certainly not
> the norm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
