Ryan Leslie created KAFKA-6287:
----------------------------------
Summary: Inconsistent protocol type for empty consumer groups
Key: KAFKA-6287
URL: https://issues.apache.org/jira/browse/KAFKA-6287
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 1.0.0
Reporter: Ryan Leslie
Priority: Minor
When a consumer is created for a new group, the group metadata's protocol type
is set to 'consumer' and this is stored both in __consumer_offsets as well as
in the coordinator's local cache.
If the consumer leaves the group and the group becomes empty, ListGroups
requests will continue to show the group as type 'consumer', and as such
kafka-consumer-groups.sh will show it via --list.
However, if the coordinator (broker) node is killed and a new coordinator is
elected, when the GroupMetadataManager loads the group from __consumer_offsets
into its cache, it will not set the protocolType if there are no active
consumers. As a result, the group's protocolType will now become the empty
string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh will no longer
show the group.
Ideally bouncing a broker should not result in the group's protocol changing.
protocolType can be set in GroupMetadataManager.readGroupMessageValue() to
always reflect what's present in the persistent metadata (__consumer_offsets)
regardless if there are active members.
In general, things can get confusing when distinguishing between 'consumer' and
non-consumer groups. For example, DescribeGroups and OffsetFetchRequest does
not filter on protocol type, so it's possible for kafka-consumer-groups.sh to
describe groups (--describe) without actually being able to list them. In the
protocol guide, OffsetFetchRequest / OffsetCommitRequest have their parameters
listed as 'ConsumerGroup', but in reality these can be used for groups of
unknown type as well. For instance, a consumer group can be copied by finding a
coordinator (GroupCoordinatorRequest / FindCoordinatorRequest) and sending an
OffsetCommitRequest. The group will be auto-created with an empty protocol.
Although this is arguably correct, the group will now exist but not be a
proper 'consumer' group until later when there is a JoinGroupRequest. Again,
this can be confusing as far as categorization / visibility of the group is
concerned. A group can instead be copied more directly by creating a consumer
and calling commitSync (as kafka-consumer-groups.sh), but this does involve
extra connections / requests and so is a little slower when trying to keep a
large number of groups in sync in real-time across clusters.
If we want to make it easier to keep consumer groups consistent, options
include allowing groups to be explicitly created with a protocol type instead
of only lazily, or for groups created outside of JoinGroupRequest the
coordinator can gain a broker config to set a default protocol type for groups.
This would default to 'consumer'. This sort of setting might be confusing for
users though, since implicitly created groups is certainly not the norm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
