This bit of info is very useful, Fabian, thank you:
You can get the parallel task id from the
> RuntimeContext.getIndexOfThisSubtask().
> RuntimeContext.getNumberOfParallelSubtasks() gives the total number of
> tasks.
---
Oytun Tez
*M O T A W O R D*
The World's Fastest Human
awesome, thanks
On Fri, 2 Aug 2019, 10:56 Fabian Hueske, wrote:
> Hi,
>
> Regarding step 3, it is sufficient to check that you got on message from
> each parallel task of the previous operator. That's because a task
> processes the timers of all keys before moving forward.
> Timers are always
Hi,
Regarding step 3, it is sufficient to check that you got on message from
each parallel task of the previous operator. That's because a task
processes the timers of all keys before moving forward.
Timers are always processed per key, but you could deduplicate on the
parallel task id and check
Hi Oytun, that sounds like a great idea thanks!! Just wanted to confirm a
couple of things:
-In step 2 by merging do you mean anything else apart from setting the
operator parallelism to 1? Forcing a parallelism of 1 should ensure all
items go to the same task.
-In step 3 I don't think I could
Perhaps:
1. collect() an item inside onTimer() inside operator#1
2. merge the resulting stream from all keys
3. process the combined stream in operator#2 to see if all keys were
processed. you will probably want to keep state in the operator#2 to see if
you received items from all
Hi all,
I have a keyed operator with an hourly event time trigger. On a timer
trigger, the operator simply persists some state to a table.
I'd like to know when the triggers for all keys have finished so I can send
a further signal to the data warehouse, to indicate it has all the
necessary data
