Re: At Least Once semantics for Kafka Streams

Ah OK! Thanks! On Mon, Feb 6, 2017, 3:09 PM Eno Thereska wrote: > Oh, by "other" I meant the original one you started discussing: > COMMIT_INTERVAL_MS_CONFIG. > > Eno > > On 6 Feb 2017, at 09:28, Mahendra Kariya > wrote: > > > > Thanks Eno! > > > > I am just wondering what is this other commit

Re: At Least Once semantics for Kafka Streams

Oh, by "other" I meant the original one you started discussing: COMMIT_INTERVAL_MS_CONFIG. Eno > On 6 Feb 2017, at 09:28, Mahendra Kariya wrote: > > Thanks Eno! > > I am just wondering what is this other commit parameter? > > On Mon, Feb 6, 2017, 12:52 PM Eno Thereska wrote: > >> Hi Mahendr

Re: At Least Once semantics for Kafka Streams

Thanks Eno! I am just wondering what is this other commit parameter? On Mon, Feb 6, 2017, 12:52 PM Eno Thereska wrote: > Hi Mahendra, > > That is a good question. Streams uses consumers and that config applies to > consumers. However, in streams we always set enable.auto.commit to false, > and

Re: At Least Once semantics for Kafka Streams

Hi Mahendra, That is a good question. Streams uses consumers and that config applies to consumers. However, in streams we always set enable.auto.commit to false, and manage commits using the other commit parameter. That way streams has more control on when offsets are committed. Eno > On 6 Fe

Re: At Least Once semantics for Kafka Streams

I have another follow up question regarding configuration. There is a config for enable.auto.commit for consumers. Does this apply to Kafka streams? If yes, how is the behavior different when the value of this config is true vs false? More generally, which of the consumer configs

Re: At Least Once semantics for Kafka Streams

Ah OK! Thanks a lot for this clarification. it will only commit the offsets if the value of COMMIT_INTERVAL_MS_CONFIG > has > passed. >

Re: At Least Once semantics for Kafka Streams

It controls the minimum frequency at which the offsets are committed. The StreamThread runs in a loop that looks something like this: while(true) records = consumer.poll(..) for each record task = findTask(record) task.process(..) end maybeCommit() end This is grossly simp

Re: At Least Once semantics for Kafka Streams

Thanks Damian for this info. On Fri, Feb 3, 2017 at 3:29 PM, Damian Guy wrote: > The commit is done on the same thread as the processing, so only offsets > that have been fully processed by the topology will be committed. > I am still not clear about why do we need the COMMIT_INTERVAL_MS_CONFI

Re: At Least Once semantics for Kafka Streams

Hey, with a little more effort you can try to make your stream application idempotent. Maybe giving you the same results. Say you want to aggregate a KStream by some key. Instead of keeping the aggregate, you keep a Set of raw values and then do the aggregate calculations with a map(). This

Re: At Least Once semantics for Kafka Streams

Hi Mahendra, The commit is done on the same thread as the processing, so only offsets that have been fully processed by the topology will be committed. Thanks, Damian On Fri, 3 Feb 2017 at 08:40 Mahendra Kariya wrote: > Thanks a lot for this Matthias. > > I have a follow up question. There is

Re: At Least Once semantics for Kafka Streams

Thanks a lot for this Matthias. I have a follow up question. There is a COMMIT_INTERVAL_MS_CONFIG config for streams. This confuses things a little bit. If the value of this config is set to, say 100 ms, does it mean that the offset will be committed after 100 ms? If yes, then how does at least on

Re: At Least Once semantics for Kafka Streams

Hi, yes, all examples have at-least-once semantics because this is the only "mode" Kafka Streams supports -- you cannot "disable" it. (btw: we are currently working on exactly-once for Streams that you will be able to turn off/on). There is not much documentation about how it work internally, bec

At Least Once semantics for Kafka Streams

Hey All, I am new to Kafka streams. From the documentation , it is pretty much clear that streams support at least once semantics. But I couldn't find details about how this is supported. I am interested in knowing the