Re: A documentation for streaming

2014-09-05 Thread Kostas Tzoumas
Looks awesome! It would also be great to have one non-trivial piece of example code in the docs that connects to a streaming source, does some transformations, and prints the output to a file (or forwards to a streaming sink). I think this is a common use case that people have in mind, so showing

Re: Scala API rewrite almost complete

2014-09-05 Thread Kostas Tzoumas
+1 I go for WebLogAnalysis. My experience with Scala consists of going through a tutorial so this will be a good stress test both for me and the new API :-) On Thu, Sep 4, 2014 at 9:09 PM, Vasiliki Kalavri vasilikikala...@gmail.com wrote: +1 for having other people implement the examples!

Re: Replacing JobManager with Scala implementation

2014-09-05 Thread Kostas Tzoumas
Hi Daniel, +1 on the argument about attracting developers being irrelevant, the argument can work both ways and is very brittle The reasons for using Akka as a library (irrespective of the programming language) have been clearly articulated in my opinion by Stephan and Till in this thread. The

Re: A documentation for streaming

2014-09-05 Thread Márton Balassi
Thanks. We're adding a join type example with some filtering, aggregating a value over a window then. Some extended version of this:

Re: Scala API rewrite almost complete

2014-09-05 Thread Márton Balassi
+1 BatchGradientDescent for me :) On Fri, Sep 5, 2014 at 11:15 AM, Kostas Tzoumas ktzou...@apache.org wrote: +1 I go for WebLogAnalysis. My experience with Scala consists of going through a tutorial so this will be a good stress test both for me and the new API :-) On Thu, Sep 4, 2014

Re: Replacing JobManager with Scala implementation

2014-09-05 Thread Till Rohrmann
Hi Daniel, the genesis of my proposal was that I resumed Asterios' effort to introduce Akka to handle our RPC calls. Asterios idea was to incorporate Akka into Flink transparently. That meant to use Akka for the MethodInvoker of the proxies and as the server to handle the calls. While rebasing it

Re: Scala API rewrite almost complete

2014-09-05 Thread Hermann Gábor
+1 ComputeEdgeDegrees for me! On Fri, Sep 5, 2014 at 11:44 AM, Márton Balassi balassi.mar...@gmail.com wrote: +1 BatchGradientDescent for me :) On Fri, Sep 5, 2014 at 11:15 AM, Kostas Tzoumas ktzou...@apache.org wrote: +1 I go for WebLogAnalysis. My experience with Scala

Community Committing rules

2014-09-05 Thread Stephan Ewen
Hi! I think part of the discussion that arose around the proposed Java/Scala and RPC/Akka changes comes from the fact that we have not clearly written down the community/committing rules anywhere yet. In particular, how do we treat proposed major changes. Most of us (including me) worked under

Re: Replacing JobManager with Scala implementation

2014-09-05 Thread Stephan Ewen
Okay, to make things clearer, I will start a separate thread for the proposed use of Akka, which talks only about the Akka/RPC question, because things keep getting confused. Let's postpone this thread (Java vs Scala) until we resolved the Akka question first. On Fri, Sep 5, 2014 at 12:10 PM,

Re: Community Committing rules

2014-09-05 Thread Márton Balassi
I'd prefer the mail vote before major changes (this is also the preferred Apache guideline if I'm not mistaken). Writing down the basics on a wiki makes it clearer and also easier for new contributors to get involved. This page is somewhat related though (at least for voting):

Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Stephan Ewen
This proposes to refactor the RPC service and the coordination between Client, JobManager, and TaskManager to use the Akka actor library. Even though Akka is written in Scala, it offers a Java interface and we can use Akka completely from Java. Below are a list of arguments why this would help

Re: Community Committing rules

2014-09-05 Thread Ufuk Celebi
+1 but I would say not the Wiki, but the How To Contribute guide. @Marton: do you have a link for the mail vote befor major changes. In any case, for me it doesn't matter whether it is a vote or a light weight mail to the dev list. On Fri, Sep 5, 2014 at 1:10 PM, Márton Balassi

Re: Replacing JobManager with Scala implementation

2014-09-05 Thread Ufuk Celebi
OK. Thanks for starting the new thread. I would say that we then [CANCEL] the ongoing vote about Scala vs. Java until the Akka issue is resolved. On Fri, Sep 5, 2014 at 12:54 PM, Stephan Ewen se...@apache.org wrote: Okay, to make things clearer, I will start a separate thread for the

Re: Community Committing rules

2014-09-05 Thread Till Rohrmann
+1 for the community committing rules. On Fri, Sep 5, 2014 at 1:39 PM, Ufuk Celebi u...@apache.org wrote: +1 but I would say not the Wiki, but the How To Contribute guide. @Marton: do you have a link for the mail vote befor major changes. In any case, for me it doesn't matter whether it is

Re: Replacing JobManager with Scala implementation

2014-09-05 Thread Till Rohrmann
I'll cancel the vote. On Fri, Sep 5, 2014 at 1:43 PM, Ufuk Celebi u...@apache.org wrote: OK. Thanks for starting the new thread. I would say that we then [CANCEL] the ongoing vote about Scala vs. Java until the Akka issue is resolved. On Fri, Sep 5, 2014 at 12:54 PM, Stephan Ewen

Re: Community Committing rules

2014-09-05 Thread Robert Metzger
I agree with Stephan: If somebody wants to do a major change and is uncertain if the community is willing to accept the change, they can ask on the mailing list about it. I would rather go with Stephan's suggestion to just drop a mail on the dev@ list, without a formal vote. If there is a

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Robert Metzger
I agree with using Akka for RPC. It is ASF 2.0 licensed, seems to have a big community [1] and users [2] that depend on the system. The YARN client is also using the old RPC service. I would like to rewrite it with Akka once we have added it into the other parts of the system, to learn it. [1]

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Kostas Tzoumas
+1 for refactoring using Akka, the arguments are overwhelming. On Fri, Sep 5, 2014 at 2:04 PM, Robert Metzger rmetz...@apache.org wrote: I agree with using Akka for RPC. It is ASF 2.0 licensed, seems to have a big community [1] and users [2] that depend on the system. The YARN client is

Re: Community Committing rules

2014-09-05 Thread Kostas Tzoumas
+1 for not requiring a vote thread unless someone asks for it On Fri, Sep 5, 2014 at 1:51 PM, Robert Metzger rmetz...@apache.org wrote: I agree with Stephan: If somebody wants to do a major change and is uncertain if the community is willing to accept the change, they can ask on the mailing

Re: Community Committing rules

2014-09-05 Thread Ufuk Celebi
I'm fine with an extra page. My initial reasoning for not going for the Wiki was to make it more promiment on the homepage. So no matter where we put it, let's make sure to have it as a sublink of the Community navigation point. On Fri, Sep 5, 2014 at 2:33 PM, Kostas Tzoumas ktzou...@apache.org

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Ufuk Celebi
+1 On Fri, Sep 5, 2014 at 2:25 PM, Kostas Tzoumas ktzou...@apache.org wrote: +1 for refactoring using Akka, the arguments are overwhelming. On Fri, Sep 5, 2014 at 2:04 PM, Robert Metzger rmetz...@apache.org wrote: I agree with using Akka for RPC. It is ASF 2.0 licensed, seems to have a

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Stephan Ewen
+1 On Fri, Sep 5, 2014 at 2:53 PM, Ufuk Celebi u...@apache.org wrote: +1 On Fri, Sep 5, 2014 at 2:25 PM, Kostas Tzoumas ktzou...@apache.org wrote: +1 for refactoring using Akka, the arguments are overwhelming. On Fri, Sep 5, 2014 at 2:04 PM, Robert Metzger rmetz...@apache.org

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Till Rohrmann
+1 On Fri, Sep 5, 2014 at 3:04 PM, Stephan Ewen se...@apache.org wrote: +1 On Fri, Sep 5, 2014 at 2:53 PM, Ufuk Celebi u...@apache.org wrote: +1 On Fri, Sep 5, 2014 at 2:25 PM, Kostas Tzoumas ktzou...@apache.org wrote: +1 for refactoring using Akka, the arguments are

Re: Scala API rewrite almost complete

2014-09-05 Thread Aljoscha Krettek
Alright, I updated my repo: https://github.com/aljoscha/incubator-flink/commits/scala-rework This now has a working WordCount example. It's pretty much a copy of the Java example with some fixups for the syntax and lambda functions. You'll also notice that I added the java-examples as a

Re: Proposal: Refactor distributed coordination to use the Akka Actor Library

2014-09-05 Thread Sebastian Schelter
+1 2014-09-05 6:46 GMT-07:00 Till Rohrmann till.rohrm...@gmail.com: +1 On Fri, Sep 5, 2014 at 3:04 PM, Stephan Ewen se...@apache.org wrote: +1 On Fri, Sep 5, 2014 at 2:53 PM, Ufuk Celebi u...@apache.org wrote: +1 On Fri, Sep 5, 2014 at 2:25 PM, Kostas Tzoumas

Re: Community Committing rules

2014-09-05 Thread Kostas Tzoumas
Henry, thank you *very much* for this input! I agree, let us not put more structure/rules than needed at this point. On Fri, Sep 5, 2014 at 6:58 PM, Henry Saputra henry.sapu...@gmail.com wrote: +1 for commits small changes directly, which is the perk of being committers of open source

[jira] [Created] (FLINK-1091) Allow joins with the solution set using key selectors

2014-09-05 Thread Vasia Kalavri (JIRA)
Vasia Kalavri created FLINK-1091: Summary: Allow joins with the solution set using key selectors Key: FLINK-1091 URL: https://issues.apache.org/jira/browse/FLINK-1091 Project: Flink Issue

[jira] [Created] (FLINK-1092) Allow operations, other than join and cogroup, on the solution set

2014-09-05 Thread Vasia Kalavri (JIRA)
Vasia Kalavri created FLINK-1092: Summary: Allow operations, other than join and cogroup, on the solution set Key: FLINK-1092 URL: https://issues.apache.org/jira/browse/FLINK-1092 Project: Flink

Re: KeySelector when joining with the solution set

2014-09-05 Thread Vasiliki Kalavri
Hi, I've created issues for (1) https://issues.apache.org/jira/browse/FLINK-1091 and (2) https://issues.apache.org/jira/browse/FLINK-1092. Regarding (3), I'm not quite sure how it should be done or whether it should be part of a more general iteration type, so I'd like to think about it a bit