Re: Rework of streaming iteration API

2015-07-07 Thread Gyula Fóra
Sorry Stephan I meant it slightly differently, I see your point: DataStream source = ... SingleInputOperator mapper = source.map(...) mapper.addInput() So the add input would be a method of the operator not the stream. Aljoscha Krettek aljos...@apache.org ezt írta (időpont: 2015. júl. 7., K,

Re: Rework of streaming iteration API

2015-07-07 Thread Gyula Fóra
@Kostas: This new API is I believe equivalent in expressivity with the current one. We can define nested loops now as well. And I also don't see nested loops much worse generally than simple loops. Gyula Fóra gyula.f...@gmail.com ezt írta (időpont: 2015. júl. 7., K, 16:14): Sorry Stephan I

Re: Rework of streaming iteration API

2015-07-07 Thread Gyula Fóra
@Aljoscha: Yes, thats basically my point as well. This is what happens now too but we give this mutable datastream a special name : IterativeDataStream This can be handled in very different ways through the api, the goal would be to make something easy to use. I am fine with what we have now

Rework of streaming iteration API

2015-07-07 Thread Gyula Fóra
Hey, Along with the suggested changes to the streaming API structure I think we should also rework the iteration api. Currently the iteration api tries to mimic the syntax of the batch API while the runtime behaviour is quite different. What we create instead of iterations is really just cyclic

Re: Rework of streaming iteration API

2015-07-07 Thread Gyula Fóra
Okay, I am fine with this approach as well I see the advantages. Then we just need to find a suitable name for marking a FeedbackPoint :) Stephan Ewen se...@apache.org ezt írta (időpont: 2015. júl. 7., K, 16:28): In Aljoscha's approach, we would need a special mutable stream. We could do it

Re: Rework of streaming iteration API

2015-07-07 Thread Paris Carbone
Good points. If we want to structured loops on streaming we will need to inject iteration counters. The question is if we really need structured iterations on plain data streams. Window iterations are must-have on the other hand... Paris On 07 Jul 2015, at 16:43, Kostas Tzoumas