[
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957204#comment-14957204
]
Joel Koshy commented on KAFKA-2017:
-----------------------------------
I agree with those benefits. If we go that route then I would prefer compaction
over other retention policies. You do lose some history, but _typically_ you
would have the last couple of state entries available if you overpartition and
time-based or size-based retention would anyway keep only a certain amount of
history. I think you can merge this in with the offsets topic (and rename it to
something like __consumer_state or something like that). If we merge then we
will end up having a heterogeneous topic with different keys
(group-topic-partition for offsets and group for state) but that should be fine.
WRT implementation complexity that was referenced above: I agree it is more
complicated to implement than ZK storage if we implement it from scratch but I
don't think we need to right? i.e., all of the fault-tolerance and caching
logic is already there in offset manager. So on coordinator failover the new
coordinator just reads from the state partitions that it now leads and loads
into memory (as we already do for offsets). Is there any other implementation
complexity that I'm missing?
> Persist Coordinator State for Coordinator Failover
> --------------------------------------------------
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Affects Versions: 0.9.0.0
> Reporter: Onur Karaman
> Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch,
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to
> a new coordinator without forcing all the consumers rejoin their groups. This
> is possible if the coordinator persists its state so that the state can be
> transferred during coordinator failover. This state consists of most of the
> information in GroupRegistry and ConsumerRegistry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
