Jason Gustafson created KAFKA-13447:
---------------------------------------
Summary: Consumer should not reuse committed offset after topic
recreation
Key: KAFKA-13447
URL: https://issues.apache.org/jira/browse/KAFKA-13447
Project: Kafka
Issue Type: Bug
Reporter: Jason Gustafson
KAFKA-12257 fixes an issue in which the consumer is unable to make progress
after a topic has been recreated. The problem was that the client could not
distinguish between stale metadata with a lower leader epoch and a recreated
topic with a lower leader epoch. With TopicId support in KIP-516, the client is
able to tell when a topic has been recreated since the new topic will have a
different ID.
However, what the patch did not fix is the potential reuse of the current
offset position on the recreated topic. For example, say that the consumer is
at offset N when the topic gets recreated. Currently, the consumer will
continue fetching from offset N after detecting the recreation. The most likely
result of this is either an offset out of range error or a log truncation
error, but it is also possible for the offset position to remain valid on the
recreated topic (say for a low-volume topic where the offsets is already low,
or a case where the consumer was down for a while).
To fix this issue completely, we need to store the topicId along with the
committed offset in __consumer_offsets. This would allow the consumer to detect
when the offset is no longer relevant for the current topic. We also need to
decide how to raise this case to the user. If the user has enabled automatic
offset reset, we can probably use that. Otherwise, we might need a new
exception type to signal the user that the position needs to be reset.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
