Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-03-02 Thread Jan Brusch
mail.com>> *Send Date:*Tue Feb 9 02:35:20 2021 *Recipients:*Jan Brusch mailto:jan.bru...@neuland-bfi.de>> *CC:*Yun Gao mailto:yungao...@aliyun.com>>, user mailto:user@flink.apache.org>> *Subject:*R

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-03-02 Thread Roman Khachatryan
For flink the state and timer are all bound to a key implicitly, thus >> I think they should >> not need to be bound manually. >> 2. To clear the outdated state, it could be cleared via count.clear(); if >> it has been 0 >> for a long time. There are different ways to coun

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-26 Thread Jan Brusch
ser mailto:user@flink.apache.org>> *Subject:*Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem Hi, Probably another solution would be to register a timer (using KeyedProcessFunction) once we see an element after

Re: Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-08 Thread Arvid Heise
mer when received the elements or update the counter to > -1, -2... to mark > how much timer it has passed. > > > Best, > Yun > > > > > --Original Mail ---------- > *Sender:*Khachatryan Roman > *Send Date:*Tue Feb 9 02:35:20 2021 > *R

Re: Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-08 Thread Yun Gao
Hi, I also think there should be different ways to achieve the target. For the first option listed previously, the pseudo-code roughly like class MyFunciton extends KeyedProcessFunction { ValueState count; void open() { count = ... // Create the value state } ​void

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-08 Thread Khachatryan Roman
Hi, Probably another solution would be to register a timer (using KeyedProcessFunction) once we see an element after keyBy. The timer will fire in windowIntervalMs. Upon firing, it will emit a dummy element which will be ignored (or subtracted) in the end. Upon receiving each new element, the

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-08 Thread Jan Brusch
Hi Yun, thanks for your reply. I do agree with your point about standard windows being for high level operations and the lower-level apis offering a rich toolset for most advanced use cases. I have tried to solve my problem with keyedProcessFunctions also but was not able to get it to work

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-07 Thread Yun Gao
Hi Jan, From my view, I think in Flink Window should be as a "high-level" operation for some kind of aggregation operation and if it could not satisfy the requirements, we could at least turn to using the "low-level" api by using KeyedProcessFunction[1]. In this case, we could use a ValueState