Hi, yes, I see operators of this style as very much an internal thing. You are probably talking about use cases for OneInputOperator and TwoInputOperator where users have a very specific need and require access the the low-level details such as watermarks, state and timers to get stuff done. Maybe we could have a wrapper for these so that users can still use them but internally we wrap them in an N-Ary Operator.
Cheers, Aljoscha On Thu, 21 Apr 2016 at 16:31 Gyula Fóra <gyf...@apache.org> wrote: > Hey, > > Some initial feedback from side: > > I think this a very important problem to deal with as a lot of applications > depend on it. I like the proposed runtime model and that is probably the > good way to handle this task, it is very clean what is happening. > > My main concern is how to handle this from the API and UDFs. What you > proposed seems like a very internal thing from the API perspective and I > would be against exposing it in the way you wrote in your example. We > should make all effort to streamline this with the functional style > operators in some way. (so in that sense the way broadcastsets are handled > is pretty nice) Maybe we could extend ds.connect() to many streams > > But in any case this is awesome initiative :) > > Cheers, > Gyula > > > Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: 2016. ápr. 21., > Cs, 15:56): > > > Hi Team, > > I'm currently thinking about how we can bring the broadcast set/broadcast > > input feature form the DataSet API to the DataStream API. I think this > > would be a valuable addition since it would enable use cases that join > > streams with constant (or slowly changing) side information. > > > > For this purpose, I think that we need to change the way we handle stream > > operators. The new model would have one unified operator that handles all > > cases and allows to add inputs after the operator was constructed, thus > > allowing the specification of broadcast inputs. > > > > I wrote up this preliminary document detailing the reason why we need > such > > a new operator for broadcast inputs and also what the API of such an > > operator would be. It also quickly touches on the required changes of > > existing per-operation stream operations such as StreamMap: > > > > > > > https://docs.google.com/document/d/1ZFzL_0xGuUEnBsFyEiHwWcmCcjhd9ArWsmhrgnt05RI/edit?usp=sharing > > > > Please have a look if you're interested. Feedback/insights are very > > welcome. :-) > > > > Cheers, > > Aljoscha > > >
