shashank created CAMEL-27317:
--------------------------------

             Summary: camel-kafka - an offset kept in the OffsetCache after a 
partition was lost, or after its commit failed on a revoke, is committed later 
and moves back the committed offset
                 Key: CAMEL-27317
                 URL: https://issues.apache.org/jira/browse/CAMEL-27317
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
            Reporter: shashank
            Assignee: shashank


h3. What happens

The Sync, Async and CommitToOffset commit managers keep the last processed 
offset of a partition in an {{OffsetCache}} until it is committed.

* {{ClassicRebalanceListener}} and {{ResumeRebalanceListener}} do not override 
{{onPartitionsLost}}, so Kafka's default calls {{onPartitionsRevoked}}, which 
commits the cached offset of a partition that another member owns already. The 
commit fails (stale generation) and the entry stays; with an 
{{offsetRepository}} the write to the repository is not refused, and the 
repository goes back.
* A revoke whose commit fails (timeout, failed async commit) keeps the entry 
too.
* {{OffsetCache.removeCommittedEntry}} removes the entry on the callback of any 
successful commit, also an older one whose callback runs after a newer offset 
was recorded (async callbacks run in a later {{poll()}} or {{commitAsync()}}).

When the partition comes back after another member committed further, and is 
revoked again (or {{breakOnFirstError}} commits it) before a record of it 
updated the entry, the stale offset is committed: the committed offset goes 
back and the other member's records are read again. In the third case a failed 
commit of the newer offset leaves nothing to commit on the revoke, and its 
records are read again.

This is not the question of CAMEL-26993 (what is recorded and committed for a 
partition the consumer still owns), only of when an entry is removed.

h3. How to reproduce

{{CommitManagerRebalanceTest}} (camel-kafka, {{MockConsumer}} with group 
offsets, refused commits and deferred async callbacks, the real 
{{ClassicRebalanceListener}}): the partition is lost or its revoke commit 
fails, another member commits 10, the partition comes back and is revoked 
again: the committed offset goes back to 5.

Found with a TLA+ model of the commit managers.

h3. Fix

New default method {{CommitManager.discardRecordedOffset}}. The rebalance 
listeners override {{onPartitionsLost}} to forget the offsets without 
committing, and forget the offsets of revoked partitions after the commit 
whether it succeeded or not. {{OffsetCache}} removes an entry after a commit 
only if the committed offset covers it.

_Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to