Re: Emit message at start and end of event time session window

Hi Till, Thank you for the explanation, I understand the behaviour now. On Thu, Mar 26, 2020 at 9:23 PM Till Rohrmann wrote: > A quick update concerning your observations. The reason why you are seeing > the unordered output is because in the gist we used > a AssignerWithPeriodicWatermarks whic

Re: Emit message at start and end of event time session window

A quick update concerning your observations. The reason why you are seeing the unordered output is because in the gist we used a AssignerWithPeriodicWatermarks which generates watermarks periodically. Due to this aspect, it can happen that Flink already process all elements up to "20" before it see

Re: Emit message at start and end of event time session window

Hmm, I might have given you a bad advice. I think the problem becomes harder because with Flink's window and trigger API we need to keep state consistent between the Trigger and the Window function. Maybe it would be easier to not rely on the windowing mechanism and instead to use Flink's process f

Re: Emit message at start and end of event time session window

Hi Till, I see, thanks for the clarification. Assuming all other setting are the same, if I generate events as follows : Element.from("1", 1000L), Element.from("2", 2000L), Element.from("3", 3000L), Element.from("10", 1L) ,Element.

Re: Emit message at start and end of event time session window

Hi Manas, the problem is that the print() statement is being executed with a different parallelism than 1. Due to this fact, the messages coming from the window function will be sent in round-robin fashion to the print operators. If you remove the setParallelism(1) from the window function, then t

Re: Emit message at start and end of event time session window

Hi Till, When I run the example code that you posted, the order of the three messages (window started, contents of window and window ended) is non-deterministic. This is surprising to me, as setParallelism(1) has been used in the pipeline - I assumed this should eliminate any form of race condition

Re: Emit message at start and end of event time session window

Great to hear that you solved the problem. Let us know if you run into any other issues. Cheers, Till On Fri, Feb 28, 2020 at 8:08 AM Manas Kale wrote: > Hi, > This problem is solved[1]. The issue was that the BroadcastStream did not > contain any watermark, which prevented watermarks for any d

Re: Emit message at start and end of event time session window

Hi, This problem is solved[1]. The issue was that the BroadcastStream did not contain any watermark, which prevented watermarks for any downstream operators from advancing. I appreciate all the help. [1] https://stackoverflow.com/questions/60430520/how-do-i-fire-downstream-oneventtime-method-when-u

Re: Emit message at start and end of event time session window

Hi Rafi and Till, Thank you for pointing out that edge case, Rafi. Till, I am trying to get this example working with the BroadcastState pattern upstream to the window operator[1]. The problem is that introducing the BroadcastState makes the onEventTime() *never* fire. Is the BroadcastState someho

Re: Emit message at start and end of event time session window

Hi Manas and Rafi, you are right that when using merging windows as event time session windows are, then Flink requires that any state the Trigger keeps is of type MergingState. This constraint allows that the state can be merged whenever two windows get merged. Rafi, you are right. With the curr

Re: Emit message at start and end of event time session window

I think one "edge" case which is not handled would be that the first event (by event-time) arrives late, then a wrong "started-window" would be reported. Rafi On Thu, Feb 20, 2020 at 12:36 PM Manas Kale wrote: > Is the reason ValueState cannot be use because session windows are always > formed

Re: Emit message at start and end of event time session window

Is the reason ValueState cannot be use because session windows are always formed by merging proto-windows of single elements, therefore a state store is needed that can handle merging. ValueState does not provide this functionality, but a ReducingState does? On Thu, Feb 20, 2020 at 4:01 PM Manas K

Re: Emit message at start and end of event time session window

Hi Till, Thanks for your answer! You also answered the next question that I was about to ask "Can we share state between a Trigger and a Window?" Currently the only (convoluted) way to share state between two operators is through the broadcast state pattern, right? Also, in your example, why can't

Re: Emit message at start and end of event time session window

Hi Manas, you can implement something like this with a bit of trigger magic. What you need to do is to define your own trigger implementation which keeps state to remember whether it has triggered the "started window" message or not. In the stateful window function you would need to do something s

Emit message at start and end of event time session window

Hi, I want to achieve the following using event time session windows: 1. When the window.getStart() and last event timestamp in the window is greater than MIN_WINDOW_SIZE milliseconds, I want to emit a message "Window started @ timestamp". 2. When the session window ends, i.e. the wate