Sounds great Fabian.
I was just trying to see if I can use higher level datastream apis.
I appreciate your advice and help.
Mans
On Monday, April 29, 2019, 5:41:36 AM EDT, Fabian Hueske
<fhue...@gmail.com> wrote:
Hi Mans,
I don't know if that would work or not. Would need to dig into the source code
for that.
TBH, I would recommend to check if you can implement the logic using a
(Keyed-)ProcessFunction.IMO, process functions are a lot easier to reason about
than Flink's windowing framework.
You can manage state and timer all by yourself and make sure everything is
properly cleaned up.
Best,Fabian
Am So., 28. Apr. 2019 um 16:31 Uhr schrieb M Singh <mans2si...@yahoo.com>:
Thanks Sameer/Rong:
As Fabian and you have mentioned, the window still sticks around forever for
global window, so I am trying avoid that scenario.
Fabian & Flink team - do you have any insights into what would happen if I
create a window and the later change it's end time during the stream processing
? Would it mess up any internal state/processing that uses the end time when
the window was first created ? If there is any other consideration to keep in
mind, please let me know.
Thanks again.
On Wednesday, April 24, 2019, 1:29:18 PM EDT, Rong Rong
<walter...@gmail.com> wrote:
Hi Mans,
Sameer is correct. if you would like to control window triggering based on
other elements that does not belong to this window (in a keyed stream context)
then this is probably the best way to approach.
I think you've also posted in another thread that describes what will be left
after fire-and-purge [1]. As Fabian stated: the only thing that might've left
after is the window (which is the 2 long values indicate the start/end) and the
trigger object. But you are right it might eventually filled up memory.
Another approach is to implement your own operator that handles all these
internally by your user code. This would require you to replicate many of the
window operator logic though.
Thanks,Rong
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-How-to-destroy-global-window-and-release-it-s-resources-td27191.html#a27212
On Wed, Apr 24, 2019 at 5:02 AM Sameer W <sam...@axiomine.com> wrote:
Global Windows is fine for this use case. I have used the same strategy. You
just define custom evictors and triggers and you are all good. Windows are
managed by keys, so as such as long as events are evicted from the window, that
counts towards reclaiming memory for the key+window combination. Plus there is
just window per key with Global Windows.
On Wed, Apr 24, 2019 at 7:47 AM M Singh <mans2si...@yahoo.com> wrote:
Hi Rong:
Thanks for your answer.
>From what I understand the dynamic gap session windows are also created when
>the event is encountered. I need to be able to change the window end time at
>a later time based on what other events are in that window. One way to do
>this is to use GlobalWindows but then these are never deleted.
Regarding CEP option - I believe that CEP patterns cannot be changed
dynamically once they've been complied which limits it usage.
Please feel free to correct me.
Thanks for your help and pointers.
On Tuesday, April 23, 2019, 8:12:56 PM EDT, Rong Rong <walter...@gmail.com>
wrote:
Hi Mans,
I am not sure what you meant by "dynamically change the end-time of a window.
If you are referring to dynamically determines the firing time of the window,
then it fits into the description of session window [1]: If you want to handle
window end time dynamically, one way of which I can think of is the dynamic
gap, session window [1] approach. with which you can specify the end-time of a
window based on input elements. Provided that you are maintaining a session
window. Another way to look at it is through the Flink-CEP library [2].
Thanks,Rong
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/windows.html#session-windows[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/libs/cep.html#groups-of-patterns
On Tue, Apr 23, 2019 at 8:19 AM M Singh <mans2si...@yahoo.com> wrote:
Hi:
I am working on a project and need to change the end time of the window
dynamically. I want to find out if the end time of the window is used
internally (for sorting windows/etc) except for handling watermarks that would
cause problems if the end time was changed during run time after the window has
been created even if no new event has arrived for that window.
I don't want to use GlobalWindow since from my understanding it never gets
destroyed.
If there is any alternate way of dealing with this, please let me know.
Thanks
Mans
