[DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-21 Thread Lin Li
Hi, When I try to implement https://issues.apache.org/jira/browse/FLINK-5498 via "dataset.coGroup(another dataset)" with a generated CoGroupFunction.(CoGroupFunction interface: public void coGroup(Iterable first, Iterable second, Collector out) I couldn't get the right results, then I sa

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Aljoscha Krettek
Hi, this is probably an oversight. If it helps you implement the feature, please go ahead and add a sub-issue for solving the Iterator problem. Best, Aljoscha On Tue, 21 Feb 2017 at 16:13 Lin Li wrote: > Hi, > > When I try to implement > https://issues.apache.org/jira/browse/FLINK-5498 > vi

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Till Rohrmann
Hi Lin Li, I think the oversight is more that we don’t throw a TraversableOnceException if you request more than one iterator as it is the case for the Iterables used for the non collection mode. Otherwise you will have a different behaviour for the collection and the non collection mode. In gene

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Ufuk Celebi
On Wed, Feb 22, 2017 at 11:19 AM, Till Rohrmann wrote: > In general, you’re right Lin Li that we don’t honour the Iterable contract > which should allow you to create an arbitrary number of iterators over the > data. Honestly, I’m not sure why we did this change because it’s not very > intuitive.

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Lin Li
Thank you for the answer! The discussion on FLINK-1023 is very clear to me. I agree with that throws a TraversableOnceException when the iterator is requested the second time. @Aljoscha git history shows you removed the exception-thrown code from FLINK-1110, would you mind me create an issue and

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Aljoscha Krettek
I think this was mostly an oversight on my part that was possible because we didn't have good test-coverage that was enforcing correctness. Please go ahead and open an issue for re-adding the throw. On Wed, 22 Feb 2017 at 13:28 Lin Li wrote: > Thank you for the answer! > > The discussion on FLIN

Re: [DISCUSS] Should we supply a new Iterator instance for Functions with Iterable input(s) like CoGroupFunction ?

2017-02-22 Thread Lin Li
I created a jira https://issues.apache.org/jira/browse/FLINK-5883, and will work on this asap. 2017-02-22 21:01 GMT+08:00 Aljoscha Krettek : > I think this was mostly an oversight on my part that was possible because > we didn't have good test-coverage that was enforcing correctness. Please go >