Re: [DISCUSS] Streaming Sources (again)

2015-05-31 Thread Stephan Ewen
I am by now also in favor of the locking variant. It pains me to say that, because I was the one pushing so heavily for the other variant ( reachedEnd() / next() ) in the first place. The concept and design of that is nice, but the reality seems to speak against it. Writing code that is properly

Re: [DISCUSS] Streaming Sources (again)

2015-05-31 Thread Márton Balassi
I am also for having only one source interface. It seems that interruptability is to much of a burden on the sources, locking version should be still acceptable from the user point of view. We are dealing with inherently concurrent tasks, I suppose our users are familiar with locking - especially

Re: [DISCUSS] Streaming Sources (again)

2015-05-31 Thread Gyula Fóra
Alright, let's do the locking then :) Let's keep only one interface for the release. On Sun, May 31, 2015 at 12:58 PM, Márton Balassi balassi.mar...@gmail.com wrote: I am also for having only one source interface. It seems that interruptability is to much of a burden on the sources, locking

[DISCUSS] Streaming Sources (again)

2015-05-29 Thread Aljoscha Krettek
Hi All, after finishing my pull request that should fix the problems with the synchronisation of checkpoints and element emission (the reason for the faulty results of the exactly-once tests) I discovered that the Kafka source does not deal well with being interrupted. We recently changed the

Re: [DISCUSS] Streaming Sources (again)

2015-05-29 Thread Gyula Fóra
Hey, It seems like both interfaces are pretty much capable of doing the same thing but work on slightly different assumptions. Isn't there a way that the kafka source can work with the interruptions? I think the reachedEnd/next interface is slightly easier to grasp than the run() with the locks.