Re: Watermark and multiple streams

[Tzu-Li (Gordon) Tai]

Hi,

How are your registering your event time timers on processElement?
If you are continuously registering them, and watermarks are correctly
generated upstream, then the onTimer method should be invoked properly.

For your 1-to-many case, I would assume that whenever a new key arrives
(that previously has not seen events with the same key from other streams),
an event time timer is registered to be fired after a certain amount of time
which you allow to wait for other matching join records.

Does this help?

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Watermark and multiple streams

[Filipe Couto]

Hello.

I'm joining several data streams, using ConnectedStreams. Let's say something 
like A connect B which outputs AB, and then I join AB with C, which outputs ABC.

However, the relationship between A and B, or AB and C may be of 1 to many, or 
1 to 1, depending on the case. For the 1 to 1, it's expected to produce an 
output as soon as I obtain both records that match the same key, but for the 1 
to many, it's not so simple, there's no event that guarantees that I have 
obtained all of my  records. For this, I've searched and found out that I 
have to implement an onTimer method, in a Process Function. I also read that an 
event timer is registered in a queue and when a new watermark arrives, the 
event timers that  will be triggered.

Given this, I'm running into trouble when generating new watermarks like in the 
example: 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-periodic-watermarks

The solution I've reached basically processes (on the onTimer method) all the 
records that I obtained previously when the first watermark arrives, but after 
that, it stops triggering.

I read that watermarks are global, so how can I create a new watermark when 
working with several streams?

Thank you