Enabling "read_committed" only ensures that a consumer does not return uncommitted data.
However, on failure, a consumer might still read committed messages multiple times (if you commit offsets after processing). If you commit offsets before you process messages, and a failure happens before processing finishes, you may "loose" those messages, as they won't be consumed again on restart. Hence, if you have a "consumer only" application, not much changed and you still need to take care in your application code about potential duplicate processing of records. -Matthias On 9/27/19 7:34 AM, Alessandro Tagliapietra wrote: > You can achieve exactly once on a consumer by enabling read committed and > manually committing the offset as soon as you receive a message. That way > you know that at next poll you won't get old message again. > > On Fri, Sep 27, 2019, 6:24 AM christopher palm <cpa...@gmail.com> wrote: > >> I had a similar question, and just watched the video on the confluent.io >> site about this. >> From what I understand idempotence and transactions are there to solve the >> duplicate writes and exactly once processing, respectively. >> >> Is what you are stating below is that this only works if we produce into a >> kafka topic and consume from it via a kafka stream, but a regular >> kafka consumer won't get the guarantee of exactly once processing? >> >> Thanks, >> Chris >> >> >> On Sat, Aug 31, 2019 at 12:29 AM Matthias J. Sax <matth...@confluent.io> >> wrote: >> >>> Exactly-once on the producer will only ensure that no duplicate writes >>> happen. If a downstream consumer fails, you might still read message >>> multiple times for all cases (ie, without idempotence, with idempotence >>> enabled, or if you use transactions). >>> >>> Note, that exactly-once is designed for a read-process-write pattern, >>> but not for a write-read pattern. >>> >>> -Matthias >>> >>> >>> >>> On 8/30/19 1:00 PM, Peter Groesbeck wrote: >>>> For a producer that emits messages to a single topic (i.e. no single >>>> message is sent to multiple topics), will enabling idempotency but not >>>> transactions provide exactly once guarantees for downstream consumers >> of >>>> said topic? >>>> >>>> Ordering is not important I just want to make sure consumers only >>> consumer >>>> messages sent once. >>>> >>> >>> >> >
signature.asc
Description: OpenPGP digital signature
