Hi, Checkpoint barriers are handled directly on top of the network layer and you are right they work similarly, by blocking input channels until it gets the barrier from all of them.
A way of implementing this on the operator level would be by adding a way to ask the inputreader the channel index of the last record. This way the operator could keep track of the channels from which it has received records and execute the watermark logic. The IndexedReaders have implemented the necessarry funcionality but were patched away accidentally buy some earlier changes (as they were not used anyway) Adding a union operator is probably an overkill and would pose the same difficulties when implementing it. Cheers, Gyula On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi Folks, > as I said in the subject. How will this work? I'm in the process about > thinking how to implement low watermarks in Streaming. I'm thinking > that the implementation should be quite similar to how the > checkpointing barriers will be implemented since they also flush out > stuff. > > Now I'm wondering how this will work with merged Streams and the > output selectors (split streams). It seems to me that there are a lot > of paths that elements can take to arrive at operators. The problem I > have is that an operator can only emit a low watermark itself if it > knows that all input operators have sent him a low watermark with that > value (the low watermark is the minimum of the low watermarks of all > upstream operators). I imagine that the checkpoint barriers exhibit > the same behaviour. > > Do we maybe have to add an explicit union (merge) operator and change > how split streams are implemented? > > What are your thoughts? > > Cheers, > Aljoscha >
