Hi folks
I want to sort stream based on event time field derived from events. To do
this I can use one of the existing windows like TimeWindow to collect
events in a window of a particular size, or SlidingWindow to run sort logic
more often (and sort within slide).
Ideally, I want to sort events a
Hi Eugene,
I'd say that what you want essentially is not "sort in windows", because
(as you mention), you want to emit elements from windows as soon as
watermark passes some timestamp. Maybe a better approach would be to
implement this using stateful processing, where you keep a buffer of
(un
Hi Jan
Thanks for a quick reply.
Doing stateful transformation requires re-writing the same logic which is
already defined in Flink by itself. Let's consider example from my original
message:
There can be out-of-order data -> data should be propagated to next
operator only when watermark crosses
Hi Eugene,
I agree with Jan. Using a ProcessFunction is the way to go.
ProcessFunction gives you all the tools you need:
* ListState which is very cheap to append to (and you only need to read the
ListState when you receive a watermark).
* Access to event timestamps, the current watermark and tim
