Jeff Kim created KAFKA-15910:
--------------------------------
Summary: New group coordinator needs to generate snapshots while
loading
Key: KAFKA-15910
URL: https://issues.apache.org/jira/browse/KAFKA-15910
Project: Kafka
Issue Type: Sub-task
Reporter: Jeff Kim
Assignee: Jeff Kim
After the new coordinator loads a __consumer_offsets partition, it logs the
following exception when making a read operation (fetch/list groups, etc):
{{{}java.lang.RuntimeException: No in-memory snapshot for epoch 740745.
Snapshot epochs are:{}}}{{{}at
org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:178){}}}{{{}at
org.apache.kafka.timeline.SnapshottableHashTable.snapshottableIterator(SnapshottableHashTable.java:407){}}}{{{}at
org.apache.kafka.timeline.TimelineHashMap$ValueIterator.<init>(TimelineHashMap.java:283){}}}{{{}at
org.apache.kafka.timeline.TimelineHashMap$Values.iterator(TimelineHashMap.java:271){}}}
{{...}}
This happens because we don't have a snapshot at the last updated high
watermark after loading. We cannot generate a snapshot at the high watermark
after loading all batches because it may contain records that have not yet been
committed. We also don't know where the high watermark will advance up to so we
need to generate a snapshot for each offset the loader observes to be greater
than the current high watermark. Then once we add the high watermark listener
and update the high watermark we can delete all of the snapshots prior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
