[
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959038#comment-14959038
]
Jiangjie Qin commented on KAFKA-2017:
-------------------------------------
My two cents.
1. About the assumption of consumer group changes is rare, I am not sure if
this is necessarily the case anymore. With new consumer, users now can
subscribe and unsubscribe from a topic dynamically. People will probably use
this feature. In that case, I would expect group change to be more frequent.
Although it depends on what information we want to persist, at least we need to
update generation ID. It might worth thinking ahead for this new use case.
2. I agree with [~jjkoshy] that most of the functionalities we need from
persisting consumer group metadata in Kafka have been implemented by offset
manager. Expanding offset manager to support more general cases of a K-V store
may have some value. Because the consumer coordinator will essenttially become
group coordinator (i.e. not only serving consumers), do we have any use case in
plan other than consumer group metadata / offsets that also use the group
coordinator? e.g. Idempotent producer?
3. Tooling might be a little cumbersome if we use Kafka to store consumer group
metadata, depending on use case.
If we are interested in a particular consumer group(in most cases we only need
this), we only need to have two round trips: one ConsumerMetadataRequest + one
GroupMetadataRequest. That is not bad.
If we want to list all the consumer groups then it become cumbersome. We need
to send something like a GroupMetadataRequest with empty group list to every
broker because only the coordinator has the group information.
4. I personally prefer to have this discussion finished before we implement the
feature instead of do zookeeper first and switch to Kafka later like we did for
offsets. But if we really want to have group metadata persisted in 0.9 and
don't want to miss the planned release again. Then like Joel said we still have
this option.
In general I still prefer to have the consumer group metadata to persist in
Kafka, mostly because it is more scalable and the coordinator needs to load the
consumer group information(group metadata + offsets) on failover anyway.
> Persist Coordinator State for Coordinator Failover
> --------------------------------------------------
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Affects Versions: 0.9.0.0
> Reporter: Onur Karaman
> Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch,
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to
> a new coordinator without forcing all the consumers rejoin their groups. This
> is possible if the coordinator persists its state so that the state can be
> transferred during coordinator failover. This state consists of most of the
> information in GroupRegistry and ConsumerRegistry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
