Yes, if you use async writes, it could lead to data loss in case if failure as offsets could have been committed before the write succeeded. Only sync writes guard you from data loss.
Note though that in Kafka Streams there is not manual commit anyway. Commits happen based on `commit.interval.ms` config. Calling `context.commit()` only schedules an earlier commit, but after the call returned, no commit happened yet (just a request to commit asap was registered). -Matthias On 7/14/21 12:00 AM, Pushkar Deole wrote: > If async-writes are used either with manual or auto commit, where the > record is sent for async write and consumer thread commits the offset > immediately, however async write fails, > > this may cause the loss of event ? > > On Wed, Jul 14, 2021 at 12:20 AM Matthias J. Sax <mj...@apache.org> wrote: > >> If you want to use EOS, you cannot use your own producer, but you need >> to use a "sink node" (via `addSink()` or `to()`). >> >> For at-least-once, if you use sync-writes, you should still get >> at-least-once guarantees. >> >> -Matthias >> >> On 7/9/21 4:12 AM, Pushkar Deole wrote: >>> Matthias, >>> >>> Do you have any inputs on above queries? >>> >>> On Wed, Jun 30, 2021 at 7:15 PM Pushkar Deole <pdeole2...@gmail.com> >> wrote: >>> >>>> Hi, >>>> >>>> Our application uses kafka streams that reads from a source topic, does >>>> processing on records and produces processed record on destination topic >>>> through the use of external producer i.e. the producer created via kafka >>>> producer API. >>>> >>>> Does this model still guarantee exactly once semantic or it won't? >>>> >>>> Currently we are using at_least_once, however the question is how >> streams >>>> handles offset commits here? >>>> Though the event is produced using synchronous API, could there be >>>> possibility of event loss in case streams commit offset before external >>>> producer hasn't produced event on destination topic yet? >>>> >>> >> >
