[ 
https://issues.apache.org/jira/browse/CAMEL-27316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-27316 started by shashank.
----------------------------------------
> camel-kafka - with batching, the records held for an incomplete batch are 
> committed after their partition was revoked, which moves back the new owner's 
> offset
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27316
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27316
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-kafka
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> h3. What happens
> With {{batching=true}} the consumer holds the records of several polls until 
> the batch is full ({{maxPollRecords}}) or {{pollTimeoutMs}} has passed 
> ({{KafkaRecordBatchingProcessor}}). When a rebalance revokes a partition 
> while some of its records are held, 
> {{ClassicRebalanceListener.onPartitionsRevoked}} only commits the 
> {{OffsetCache}}; the held records stay in the batch. The batch is dispatched 
> later, and its {{CommitSynchronization}} commits the revoked partition 
> ({{recordOffset}} + {{commit(partition)}}):
> # consumer A polls records 0 to 2 of partition P and holds them;
> # a rebalance gives P to consumer B, which reads from the committed offset 
> (0), processes up to 9 and commits 10;
> # A's batch is dispatched: A processes 0 to 2 again and commits 3. The 
> committed offset goes back from 10 to 3, and records 3 to 9 are read again by 
> the next owner.
> This happens with the default options: with batching, 
> {{KafkaConfiguration.isAutoCommitEnable()}} is false (the client does not 
> auto commit) and Camel's {{AsyncCommitManager}} commits each batch. The 
> classic group protocol checks the generation of a commit, not which 
> partitions the member owns, so the commit is accepted. With an eager 
> rebalance the same consumer can also get P back, read 0 to 2 again, and hold 
> them twice in one batch.
> CAMEL-26978 handled the same case for asynchronous manual commits (a commit 
> requested after the partition was revoked is not committed).
> h3. How to reproduce
> {{KafkaBatchingConsumerRevokeTest}} (camel-kafka, {{MockConsumer}}): hold 3 
> records of partition 0, revoke it, let the new owner commit 10, then fill the 
> batch with records of partition 1: the committed offset of partition 0 goes 
> back to 3 and the batch holds the records of the revoked partition.
> Found with a TLA+ model of the batching consumer.
> h3. Fix
> On a revoke (or loss) of partitions, the rebalance listener wrapper of 
> {{KafkaFetchRecords}} asks the record processor to drop the held records of 
> those partitions ({{KafkaRecordProcessorFacade.discardPendingRecords}}, a new 
> default method; the batching facade removes and releases the exchanges). The 
> next owner reads them from the committed offset.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to