[
https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-6287.
------------------------------------
Resolution: Fixed
> Inconsistent protocol type for empty consumer groups
> ----------------------------------------------------
>
> Key: KAFKA-6287
> URL: https://issues.apache.org/jira/browse/KAFKA-6287
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 1.0.0
> Reporter: Ryan Leslie
> Assignee: Jason Gustafson
> Priority: Minor
> Fix For: 1.0.1
>
>
> When a consumer is created for a new group, the group metadata's protocol
> type is set to 'consumer' and this is stored both in __consumer_offsets as
> well as in the coordinator's local cache.
> If the consumer leaves the group and the group becomes empty, ListGroups
> requests will continue to show the group as type 'consumer', and as such
> kafka-consumer-groups.sh will show it via --list.
> However, if the coordinator (broker) node is killed and a new coordinator is
> elected, when the GroupMetadataManager loads the group from
> __consumer_offsets into its cache, it will not set the protocolType if there
> are no active consumers. As a result, the group's protocolType will now
> become the empty string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh
> will no longer show the group.
> Ideally bouncing a broker should not result in the group's protocol changing.
> protocolType can be set in GroupMetadataManager.readGroupMessageValue() to
> always reflect what's present in the persistent metadata (__consumer_offsets)
> regardless if there are active members.
> In general, things can get confusing when distinguishing between 'consumer'
> and non-consumer groups. For example, DescribeGroups and OffsetFetchRequest
> does not filter on protocol type, so it's possible for
> kafka-consumer-groups.sh to describe groups (--describe) without actually
> being able to list them. In the protocol guide, OffsetFetchRequest /
> OffsetCommitRequest have their parameters listed as 'ConsumerGroup', but in
> reality these can be used for groups of unknown type as well. For instance, a
> consumer group can be copied by finding a coordinator
> (GroupCoordinatorRequest / FindCoordinatorRequest) and sending an
> OffsetCommitRequest. The group will be auto-created with an empty protocol.
> Although this is arguably correct, the group will now exist but not be a
> proper 'consumer' group until later when there is a JoinGroupRequest. Again,
> this can be confusing as far as categorization / visibility of the group is
> concerned. A group can instead be copied more directly by creating a consumer
> and calling commitSync (as kafka-consumer-groups.sh), but this does involve
> extra connections / requests and so is a little slower when trying to keep a
> large number of groups in sync in real-time across clusters.
> If we want to make it easier to keep consumer groups consistent, options
> include allowing groups to be explicitly created with a protocol type instead
> of only lazily, or for groups created outside of JoinGroupRequest the
> coordinator can gain a broker config to set a default protocol type for
> groups. This would default to 'consumer'. This sort of setting might be
> confusing for users though, since implicitly created groups is certainly not
> the norm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
