Re: State flush & recovery during failures

2018-09-23 Thread Matthias J. Sax
Not exactly sure what you mean by "restored"? In general, there are two types of punctuation: wall-clock time and event-time. For wall-clock time, it's somewhat non-deterministic when they will be called. For event-time, as those depend on the immutable input, they will be called again as in the

Re: State flush & recovery during failures

2018-09-21 Thread Vishnu Viswanath
Thank you Matthias, That cleared things up, have one follow up question on punctuations. Does punctuations get restored after failure? If not what is the right way to do so? Is it by storing the info currently active punctuations in some state and by scheduling them again in the init method? Tha

Re: State flush & recovery during failures

2018-09-20 Thread Matthias J. Sax
1. The record will be re-read, but the state would not be re-build (ie, no undo of step (2). Thus, on re-processing you would add the record again, and you would "over count" in step (3) -- trigger would still fire I assume. 2. I assume, by "forward" you mean writing to an output topic: If you wri

State flush & recovery during failures

2018-09-18 Thread Vishnu Viswanath
Hi All, I have KafkaStreams application (processor API) that does roughly the below steps. 1. read 2. add to state 3. check state size (count based trigger) 3.1 process 3.2 delete records from state 3.3 forward 3.4 commit (kafka internally does) 3.4.1 flush state 3.4.2 flush producer 3.4.3 commit