[ https://issues.apache.org/jira/browse/CAMEL-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-14935. --------------------------------- Resolution: Implemented > KafkaConsumer commits old offset values in a failure scenario causing message > replays and offset reset error > ------------------------------------------------------------------------------------------------------------ > > Key: CAMEL-14935 > URL: https://issues.apache.org/jira/browse/CAMEL-14935 > Project: Camel > Issue Type: Bug > Components: camel-kafka > Affects Versions: 2.24.0 > Reporter: Chris McCarthy > Priority: Major > Fix For: 3.x > > > We are experiencing unexpected offset reset errors occasionally, as well as > occasional replay of messages (without an offset reset error). > The cause seems to be a failed commit on rebalance, leaving an old value in > the hashMap used to store the latest processed offset for a partition. This > old value is then re-read and re-committed across rebalances in certain > situations. > Our relevant configuration details are: > autoCommitEnable=false > allowManualCommit=true > autoOffsetReset=earliest > It seems when the KafkaConsumer experiences an Exception committing the > offset (CommitFailedException) upon a rebalance, this leaves the old offset > value in the lastProcessedOffset hashMap. > A subsequent rebalance that assigns the same partition to the same consumer, > that then thereafter experiences another rebalance (before any messages have > been processed successfully as this will over write the invalid value and > self correct the problem) will commit this old offset again. This offset may > be very old if there have been many rebalances in between the original > rebalance that failed to commit its offset. > If the old offset is beyond the retention period and the message is no longer > available the outcome is an offset reset error. If the offset is within the > retention period all messages are replayed from that offset without an error > being thrown. -- This message was sent by Atlassian Jira (v8.3.4#803005)