Re: [DISCUSS] Graduation of Flink from the Incubator

2014-11-27 Thread Stephan Ewen
Hi! I feel very excited about this idea. From what I can see, the community has acted very mature. Releases and community growth are happening as well. The community is diversifying and the project's life and death no longer hinges on single individuals. Stephan Am 28.11.2014 01:27 schrieb "Fabia

Re: [DISCUSS] Graduation of Flink from the Incubator

2014-11-27 Thread Fabian Hueske
I agree with Kostas. Let's go for the graduation! Cheers, Fabian 2014-11-27 20:34 GMT+01:00 Kostas Tzoumas : > Hi everyone, > > I would like to share a discussion that we have been having on the PPMC > list. > > Flink has successfully made two Apache releases and added 5 committers > since enter

Re: New Flink website layout and frontage

2014-11-27 Thread Fabian Hueske
Thanks for putting this together. Looks really good! IMO, the tag line should make clear that Flink is a system for parallel data processing. 2014-11-27 23:35 GMT+01:00 Markl, Volker, Prof. Dr. < volker.ma...@tu-berlin.de>: > +1 > > Von meinem iPhone gesendet > > > Am 27.11.2014 um 20:37 schrie

Re: New Flink website layout and frontage

2014-11-27 Thread Markl, Volker, Prof. Dr.
+1 Von meinem iPhone gesendet > Am 27.11.2014 um 20:37 schrieb "Gyula Fora" : > > +1 > > I also think that at looks very good! > Stylish :) > >> On 27 Nov 2014, at 20:32, Stephan Ewen wrote: >> >> I like it as well. Much better than the current website :-) >> >> On Thu, Nov 27, 2014 at 8:30

Re: New Flink website layout and frontage

2014-11-27 Thread Kirschnick, Johannes
I also like the shorter tag line. What about using 3 attributes.. As three is a good number and it also preserves the layout that the text is beneath and both sides of the logo Like Fast, reliable, large scale data processing. Or with clusters.. It should be at least "on clusters" 😁 Johanne

Re: New Flink website layout and frontage

2014-11-27 Thread Kirschnick, Johannes
I also like the design.. Looks almost too good for an open source project 😁 Some minor things Shouldn't it be run on Yarn as contrary to run in Yarn? Also the section about reliable and scalable. This could benefit from a bit of rewording.. Guess all three paragraphs should mention the word FL

Re: New Flink website layout and frontage

2014-11-27 Thread Max Michels
+1 The new website presents a much better overview for the general user. On Thu, Nov 27, 2014 at 8:41 PM, Vasiliki Kalavri wrote: > Hey! > > It looks great :)) > > If I would only change one thing, I would remove "in clusters" from the > tagline. > First, because it can also be locally and in t

Re: New Flink website layout and frontage

2014-11-27 Thread Vasiliki Kalavri
Hey! It looks great :)) If I would only change one thing, I would remove "in clusters" from the tagline. First, because it can also be locally and in the cloud and second, because the shorter the tagline the better, no? ;) Cheers, V. On 27 November 2014 at 20:34, Gyula Fora wrote: > +1 > > I

Re: New Flink website layout and frontage

2014-11-27 Thread Gyula Fora
+1 I also think that at looks very good! Stylish :) > On 27 Nov 2014, at 20:32, Stephan Ewen wrote: > > I like it as well. Much better than the current website :-) > > On Thu, Nov 27, 2014 at 8:30 PM, Aljoscha Krettek > wrote: > >> The formatting of the Scala example code is a bit off in pla

[DISCUSS] Graduation of Flink from the Incubator

2014-11-27 Thread Kostas Tzoumas
Hi everyone, I would like to share a discussion that we have been having on the PPMC list. Flink has successfully made two Apache releases and added 5 committers since entering the Incubator. The community is openly discussing in the mailing lists, and new contributors and users are coming to the

Re: New Flink website layout and frontage

2014-11-27 Thread Stephan Ewen
I like it as well. Much better than the current website :-) On Thu, Nov 27, 2014 at 8:30 PM, Aljoscha Krettek wrote: > The formatting of the Scala example code is a bit off in places. Other > than that it looks very good. > > On Thu, Nov 27, 2014 at 8:22 PM, Kostas Tzoumas > wrote: > > Actually

Re: New Flink website layout and frontage

2014-11-27 Thread Aljoscha Krettek
The formatting of the Scala example code is a bit off in places. Other than that it looks very good. On Thu, Nov 27, 2014 at 8:22 PM, Kostas Tzoumas wrote: > Actually, I cancelled the commit. > > Perhaps this is a good time to comment on the new frontpage (the rest of > the content is the same).

[jira] [Created] (FLINK-1292) Allow for longer normalized keys

2014-11-27 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-1292: --- Summary: Allow for longer normalized keys Key: FLINK-1292 URL: https://issues.apache.org/jira/browse/FLINK-1292 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-1291) Remove GC options from JobManager and TaskManager scripts

2014-11-27 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-1291: --- Summary: Remove GC options from JobManager and TaskManager scripts Key: FLINK-1291 URL: https://issues.apache.org/jira/browse/FLINK-1291 Project: Flink Issue T

Re: New Flink website layout and frontage

2014-11-27 Thread Kostas Tzoumas
Actually, I cancelled the commit. Perhaps this is a good time to comment on the new frontpage (the rest of the content is the same). You can see the frontage in pdf (two parts, as you would scroll down) here: https://www.dropbox.com/sh/4b9f4judlczfzkp/AADIsdw1vxyNVZH6oyWsAIWNa?dl=0 I think we ca

New Flink website layout and frontage

2014-11-27 Thread Kostas Tzoumas
Hi everyone, Ufuk, Robert, and I worked on a new layout for the Flink website, and added some content to the frontpage to (hopefully) convey more clearly what Flink is all about and what are Flink's differentiating features (a point of criticism to the old website was often that it is very non-des

Re: Samoa on Flink

2014-11-27 Thread Márton Balassi
Hey, Thanks Vasia, great idea. When we've started the streaming development the first sketch API we had is was actually really similar to Storm's API a couple of versions back. I've talked to Gyula about the issue when we've first heard about SAMOA ago and we are confident that porting Samoa to Fl

Re: Compression of network traffic

2014-11-27 Thread Stephan Ewen
Hey! As an example, you can have a look at the normalized key sorter, which sorts and compares complete on serialized data. Stephan On Thu, Nov 27, 2014 at 2:25 PM, Viktor Rosenfeld < viktor.rosenf...@tu-berlin.de> wrote: > Hi Stephan, > > can you give me an example (or a few) where Flink is w

Re: Compression of network traffic

2014-11-27 Thread Viktor Rosenfeld
Hi Stephan, can you give me an example (or a few) where Flink is working on the serialized data? Cheers, Viktor -- View this message in context: http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Compression-of-network-traffic-tp2568p2651.html Sent from the Apache Flink

Re: Samoa on Flink

2014-11-27 Thread Paris Carbone
Hey Vasia, I really like the idea and it seems we have some volunteers from our side already! It is in my opinion a starter task since it requires to create a simple adapter for flink as well as mapping abstractions to our streaming topology API in a similar fashion (in its simplest form just u

Samoa on Flink

2014-11-27 Thread Vasiliki Kalavri
Hello everyone, Last week at Strata, I attended a talk about a streaming machine learning framework from Yahoo labs, SAMOA: https://github.com/yahoo/samoa. The project is going through apache incubation application and currently runs on Storm, S4 and Samza. After the talk, I spoke with the guy th

[jira] [Created] (FLINK-1290) Optimizer prunes all candidates when unable to reuse sort properties

2014-11-27 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-1290: --- Summary: Optimizer prunes all candidates when unable to reuse sort properties Key: FLINK-1290 URL: https://issues.apache.org/jira/browse/FLINK-1290 Project: Flink

Re: Hi / Aggregation support

2014-11-27 Thread Aljoscha Krettek
Ahh, I didn't see that. My bad. On Thu, Nov 27, 2014 at 11:47 AM, Fabian Hueske wrote: > Viktor said he changed the implementation to > MapFunction -> ReduceFunction -> MapFunction. > > So it is combinable :-) > > 2014-11-27 11:45 GMT+01:00 Aljoscha Krettek : > >> Hi, >> why does the GroupReduce

Re: Hi / Aggregation support

2014-11-27 Thread Fabian Hueske
Hi Viktor, thanks for the update! Regarding the explicit vs implicit adding of key fields: I would only allow to use either key(x) or allKeys() and throw an exception if they are mixed. I guess there won't be many situations, where somebody would want to mix them anyway. No need to complicate the

Re: Hi / Aggregation support

2014-11-27 Thread Fabian Hueske
Viktor said he changed the implementation to MapFunction -> ReduceFunction -> MapFunction. So it is combinable :-) 2014-11-27 11:45 GMT+01:00 Aljoscha Krettek : > Hi, > why does the GroupReduce change the output type? Can this not be done > in the two mappers? In my opinion, aggregations should

Re: Hi / Aggregation support

2014-11-27 Thread Aljoscha Krettek
Hi, why does the GroupReduce change the output type? Can this not be done in the two mappers? In my opinion, aggregations should be combinable, otherwise, performance would be severely crippled. Cheers, Aljoscha On Thu, Nov 27, 2014 at 11:20 AM, Viktor Rosenfeld wrote: > Hi Fabian, > > thanks fo

Re: Hi / Aggregation support

2014-11-27 Thread Viktor Rosenfeld
Hi Fabian, thanks for your feedback. See my responses below. Fabian Hueske wrote > - I would split the branch into two branches, one for each approach. That > make comparisons with master much easier. I've moved the changes necessary for the second approach to a branch called aggregation-alt: h

Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Stephan Ewen
Our implication so far was that forwarding means evenly scattering over successors - a balanced load being the important goal. If you find different requirements in streaming, you could define a new type of selector. On Thu, Nov 27, 2014 at 11:02 AM, Gyula Fora wrote: > Thanks Stephan, > > So I

Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Gyula Fora
Thanks Stephan, So I took a quick look at the ChannelSelectors the batch api uses and I see that for Forward strategy uses round-robin. My question was aimed exactly to avoid having to do this. Isn’t this sub-optimal? Maybe we could pass the channel info to the channel selector, so it can make

Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Stephan Ewen
This is a bit tricky, since the new scheduling is more flexible... Assume we have a PointWise connection with two receiving tasks per sending task: outgoing channels 0 and 1. When scheduling in the most basic mode, the receivers can go anywhere, but the schedule will try to give them a slot on th

Re: Compression of network traffic

2014-11-27 Thread Stephan Ewen
Yes, that working on serialized data happens in parts right now and it would be great to extend that. While it would be possible to work on a compact serialized representation, I can't think of a way to work on a snappy/lz4 compressed version. Am 26.11.2014 23:00 schrieb "Viktor Rosenfeld" < vikto