Chris Egerton created KAFKA-13472:
-------------------------------------
Summary: Connect can lose track of last committed offsets for
topic partitions after partial consumer revocation
Key: KAFKA-13472
URL: https://issues.apache.org/jira/browse/KAFKA-13472
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.1.0, 3.0.1
Reporter: Chris Egerton
Assignee: Chris Egerton
The Connect framework tracks the last successfully-committed offsets for each
topic partition that is currently assigned to the consumer of each sink task.
If a sink task throws an exception from {{{}SinkTask::preCommit{}}}, the
consumer is "rewound" by seeking to those last successfully-committed offsets
for each topic partition, so that the same records can be redelivered to the
task again.
With the changes from KAFKA-12487, we failed to correctly update the logic for
tracking these last-committed offsets which can cause topic partitions to be
missing from them after partial revocation of topic partitions from the
consumer. Specifically, we make the assumption that, whenever an offset commit
succeeds, the offsets that were successfully committed constitute the entirely
of the last-committed offsets for the task; when a partial consumer revocation
takes place, we only commit offsets for some of the topic partitions assigned
to the task's producer, and this assumption fails.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
