The app cannot know. That is why you need to use sync-writes. Kafka Streams won't commit offset at long as custom code is executed, thus, if you call `producer.send()` and wait to the ack and block within `Processor.process()` you can be sure that no commit happens in between, ie., you can be sure that yoru write will succeed, before the next commit may happen.
Yes, there can be multiple sink nodes. -Matthias On 7/14/21 10:09 PM, Pushkar Deole wrote: > That's exactly what my question was: since there is an external producer in > the application without a sink node in topology, how will the streams know > that task is completed before committing the offset or it will not know at > all ? > > Second question is: can there be multiple sink nodes if the record is to be > produced on different topics based on some conditions (e.g. just giving a > simplified example: if record contains field A then produce on topic A, > field B then produce on Topic B etc.) > > On Thu, Jul 15, 2021 at 4:29 AM Matthias J. Sax <mj...@apache.org> wrote: > >> Yes, if you use async writes, it could lead to data loss in case if >> failure as offsets could have been committed before the write succeeded. >> Only sync writes guard you from data loss. >> >> Note though that in Kafka Streams there is not manual commit anyway. >> Commits happen based on `commit.interval.ms` config. Calling >> `context.commit()` only schedules an earlier commit, but after the call >> returned, no commit happened yet (just a request to commit asap was >> registered). >> >> >> -Matthias >> >> On 7/14/21 12:00 AM, Pushkar Deole wrote: >>> If async-writes are used either with manual or auto commit, where the >>> record is sent for async write and consumer thread commits the offset >>> immediately, however async write fails, >>> >>> this may cause the loss of event ? >>> >>> On Wed, Jul 14, 2021 at 12:20 AM Matthias J. Sax <mj...@apache.org> >> wrote: >>> >>>> If you want to use EOS, you cannot use your own producer, but you need >>>> to use a "sink node" (via `addSink()` or `to()`). >>>> >>>> For at-least-once, if you use sync-writes, you should still get >>>> at-least-once guarantees. >>>> >>>> -Matthias >>>> >>>> On 7/9/21 4:12 AM, Pushkar Deole wrote: >>>>> Matthias, >>>>> >>>>> Do you have any inputs on above queries? >>>>> >>>>> On Wed, Jun 30, 2021 at 7:15 PM Pushkar Deole <pdeole2...@gmail.com> >>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Our application uses kafka streams that reads from a source topic, >> does >>>>>> processing on records and produces processed record on destination >> topic >>>>>> through the use of external producer i.e. the producer created via >> kafka >>>>>> producer API. >>>>>> >>>>>> Does this model still guarantee exactly once semantic or it won't? >>>>>> >>>>>> Currently we are using at_least_once, however the question is how >>>> streams >>>>>> handles offset commits here? >>>>>> Though the event is produced using synchronous API, could there be >>>>>> possibility of event loss in case streams commit offset before >> external >>>>>> producer hasn't produced event on destination topic yet? >>>>>> >>>>> >>>> >>> >> >
