The app cannot know. That is why you need to use sync-writes. Kafka
Streams won't commit offset at long as custom code is executed, thus, if
you call `producer.send()` and wait to the ack and block within
`Processor.process()` you can be sure that no commit happens in between,
ie., you can be sure
That's exactly what my question was: since there is an external producer in
the application without a sink node in topology, how will the streams know
that task is completed before committing the offset or it will not know at
all ?
Second question is: can there be multiple sink nodes if the
Yes, if you use async writes, it could lead to data loss in case if
failure as offsets could have been committed before the write succeeded.
Only sync writes guard you from data loss.
Note though that in Kafka Streams there is not manual commit anyway.
Commits happen based on `commit.interval.ms`
If async-writes are used either with manual or auto commit, where the
record is sent for async write and consumer thread commits the offset
immediately, however async write fails,
this may cause the loss of event ?
On Wed, Jul 14, 2021 at 12:20 AM Matthias J. Sax wrote:
> If you want to use
If you want to use EOS, you cannot use your own producer, but you need
to use a "sink node" (via `addSink()` or `to()`).
For at-least-once, if you use sync-writes, you should still get
at-least-once guarantees.
-Matthias
On 7/9/21 4:12 AM, Pushkar Deole wrote:
> Matthias,
>
> Do you have any
Matthias,
Do you have any inputs on above queries?
On Wed, Jun 30, 2021 at 7:15 PM Pushkar Deole wrote:
> Hi,
>
> Our application uses kafka streams that reads from a source topic, does
> processing on records and produces processed record on destination topic
> through the use of external
Hi,
Our application uses kafka streams that reads from a source topic, does
processing on records and produces processed record on destination topic
through the use of external producer i.e. the producer created via kafka
producer API.
Does this model still guarantee exactly once semantic or it
