Hi Mahendra, The commit is done on the same thread as the processing, so only offsets that have been fully processed by the topology will be committed.
Thanks, Damian On Fri, 3 Feb 2017 at 08:40 Mahendra Kariya <mahendra.kar...@go-jek.com> wrote: > Thanks a lot for this Matthias. > > I have a follow up question. There is a COMMIT_INTERVAL_MS_CONFIG config > for streams. This confuses things a little bit. If the value of this config > is set to, say 100 ms, does it mean that the offset will be committed after > 100 ms? If yes, then how does at least once guarantee work? > > > > > On Mon, Jan 30, 2017 at 1:43 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > Hi, > > > > yes, all examples have at-least-once semantics because this is the only > > "mode" Kafka Streams supports -- you cannot "disable" it. (btw: we are > > currently working on exactly-once for Streams that you will be able to > > turn off/on). > > > > There is not much documentation about how it work internally, because it > > uses the same mechanism Kafka consumer user to provide at-least-once > > semantics and there is good documentation for this: > > http://docs.confluent.io/current/clients/consumer.html > > > > The basic idea is, that input topic offsets get only committed after the > > processing is completely finished and all intermediate state/data is > > flushed to reliable storage -- to ensure nothing can get lost. If a > > failure happens, just retry from last committed offset (and because some > > output date might already be written which cannot be undone you might > > get duplicates). > > > > -Matthias > > > > On 1/29/17 8:13 PM, Mahendra Kariya wrote: > > > Hey All, > > > > > > I am new to Kafka streams. From the documentation > > > <http://docs.confluent.io/3.1.0/streams/architecture.html# > > processing-guarantees>, > > > it is pretty much clear that streams support at least once semantics. > > But I > > > couldn't find details about how this is supported. I am interested in > > > knowing the finer details / design of this. > > > > > > Is there some documentation around this? > > > Is there some documentation around what semantics are followed by the > > > various Kafka streams examples > > > <https://github.com/confluentinc/examples/tree/3.1.x/kafka-streams> > > > available on Github? Do all of them follow at least once? > > > > > > > > > Thanks, > > > Mahendra > > > > > > > >
