Re: Introducing starter tag to indicate easy JIRAs

2014-06-24 Thread Henry Saputra
Cool, if no objection I will start tagging the JIRAs that Fabian's had sent out as starter issues. Thanks, Henry On Tue, Jun 24, 2014 at 8:40 AM, Markus Holzemer wrote: > +1 > Good Idea! > Am 24.06.2014 17:27 schrieb "Henry Saputra" : > >> HI All, >> >> Continuing Fabian's response for some eas

[jira] [Created] (FLINK-980) Buffer leak in OutputChannel#sendBuffer(Buffer)

2014-06-24 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-980: - Summary: Buffer leak in OutputChannel#sendBuffer(Buffer) Key: FLINK-980 URL: https://issues.apache.org/jira/browse/FLINK-980 Project: Flink Issue Type: Bug

[GitHub] incubator-flink pull request: [FLINK-979] Fix NetworkThroughput te...

2014-06-24 Thread uce
GitHub user uce opened a pull request: https://github.com/apache/incubator-flink/pull/41 [FLINK-979] Fix NetworkThroughput test input and output task config - Set DummyInputFormat and DummyOutputFormat via TaskConfig to respect task hierarchy refactoring. - Run test via main met

[jira] [Created] (FLINK-979) Scheduler and JobGrph refactoring broke NetworkThroughput test

2014-06-24 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-979: - Summary: Scheduler and JobGrph refactoring broke NetworkThroughput test Key: FLINK-979 URL: https://issues.apache.org/jira/browse/FLINK-979 Project: Flink Issue T

[jira] [Created] (FLINK-978) JobManager Webfrontend does not display running jobs since switch to slot-based Scheduler

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-978: -- Summary: JobManager Webfrontend does not display running jobs since switch to slot-based Scheduler Key: FLINK-978 URL: https://issues.apache.org/jira/browse/FLINK-978 Pro

[jira] [Created] (FLINK-977) Wrong error message when TaskManager cannot connect to Jobmanager

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-977: -- Summary: Wrong error message when TaskManager cannot connect to Jobmanager Key: FLINK-977 URL: https://issues.apache.org/jira/browse/FLINK-977 Project: Flink Is

[jira] [Created] (FLINK-976) When TaskManager fails to connect to teh JobManager

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-976: -- Summary: When TaskManager fails to connect to teh JobManager Key: FLINK-976 URL: https://issues.apache.org/jira/browse/FLINK-976 Project: Flink Issue Type: Impro

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Henry Saputra
+1 I am in Might as well share the Hangout link in the dev@ list just in case people would like to drop by - Henry On Tue, Jun 24, 2014 at 6:09 AM, Stephan Ewen wrote: > @everyone interested in the Tez work: > > I created a JIRA Issue with a brief summary of the current status and > plans: htt

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Henry Saputra
Thanks for the explanation Kostas. I am hoping to keep the Flink APIs (i.e. the operator functions) clean and hide all Tez nitty gritty in the plan execution =) - Henry On Tue, Jun 24, 2014 at 5:05 AM, Kostas Tzoumas wrote: > Henry, > > I am currently travelling and be able to write more about

[jira] [Created] (FLINK-975) Show the connected TaskManagers in the JobManager web frontend

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-975: -- Summary: Show the connected TaskManagers in the JobManager web frontend Key: FLINK-975 URL: https://issues.apache.org/jira/browse/FLINK-975 Project: Flink Issue

Re: Clearing the old Stratosphere Repository

2014-06-24 Thread Stephan Ewen
Sounds good!

Re: Clearing the old Stratosphere Repository

2014-06-24 Thread Henry Saputra
Hi Stephan, For now we could add huge warning message in the stratosphere's Github readme file to indicate it is deprecated and already move to ASF incubator git (via its Github mirror). - Henry On Tue, Jun 24, 2014 at 9:00 AM, Stephan Ewen wrote: > At some point we should we clear the old stra

[GitHub] incubator-flink pull request: Local Executor pick up the plans req...

2014-06-24 Thread StephanEwen
Github user StephanEwen commented on the pull request: https://github.com/apache/incubator-flink/pull/40#issuecomment-47002309 Extended by adding environment logging (sorry, I oushed to the wrong branch, so it ended up here). JobManager and TaskManager log teh following upon s

[jira] [Created] (FLINK-974) TaskManager startup script does not respect JAVA_HOME

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-974: -- Summary: TaskManager startup script does not respect JAVA_HOME Key: FLINK-974 URL: https://issues.apache.org/jira/browse/FLINK-974 Project: Flink Issue Type: New

[GitHub] incubator-flink pull request: Local Executor pick up the plans req...

2014-06-24 Thread StephanEwen
GitHub user StephanEwen opened a pull request: https://github.com/apache/incubator-flink/pull/40 Local Executor pick up the plans required degree parallelism. This is a crucial fix for the local mode in the slot based model. Please review soon. You can merge this pull request into

Clearing the old Stratosphere Repository

2014-06-24 Thread Stephan Ewen
At some point we should we clear the old stratosphere github repository (delete all files except Readme) and change the readme to point to the apache/incubator-flink repository.

Re: Introducing starter tag to indicate easy JIRAs

2014-06-24 Thread Markus Holzemer
+1 Good Idea! Am 24.06.2014 17:27 schrieb "Henry Saputra" : > HI All, > > Continuing Fabian's response for some easy JIRAs for new contributors, > I would like to introduce "starter" tag to indicate Flink's JIRAs that > supposedly isolated and good start into the project. > > This usually helps ne

Introducing starter tag to indicate easy JIRAs

2014-06-24 Thread Henry Saputra
HI All, Continuing Fabian's response for some easy JIRAs for new contributors, I would like to introduce "starter" tag to indicate Flink's JIRAs that supposedly isolated and good start into the project. This usually helps newcomers to project to get their feet wet and feel welcomed for their cont

[jira] [Created] (FLINK-973) Remove printing config to System.out

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-973: -- Summary: Remove printing config to System.out Key: FLINK-973 URL: https://issues.apache.org/jira/browse/FLINK-973 Project: Flink Issue Type: New Feature

Re: Runtime exception in Delta Iteration

2014-06-24 Thread Stephan Ewen
The delta iterations are currently still memory sensitive (the last component in flink that is). We are working on providing better memory usage for them, but the only short term solution is to increase the memory size of the JVMs that run the code, or to use more machines. Stephan On Tue, Jun

Runtime exception in Delta Iteration

2014-06-24 Thread Janani Chakkaradhari
Hi, I am facing the following exception while using Delta iteration. The dataset size is 27GB. Any suggestions to overcome this issue? 15:14:30,431 ERROR eu.stratosphere.pact.runtime.task.RegularPactTask - Error in task code: SolutionSet Delta (142/284) java.lang.RuntimeException: Me

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Fabian Hueske
I'm in as well. 2014-06-24 16:35 GMT+02:00 Kostas Tzoumas : > Stephan, great work, thank you! I am interested > > Kostas > > > On Tue, Jun 24, 2014 at 3:09 PM, Stephan Ewen wrote: > > > @everyone interested in the Tez work: > > > > I created a JIRA Issue with a brief summary of the current stat

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Kostas Tzoumas
Stephan, great work, thank you! I am interested Kostas On Tue, Jun 24, 2014 at 3:09 PM, Stephan Ewen wrote: > @everyone interested in the Tez work: > > I created a JIRA Issue with a brief summary of the current status and > plans: https://issues.apache.org/jira/browse/FLINK-972 > > I was think

[GitHub] incubator-flink pull request: Documentation update markdown

2014-06-24 Thread skunert
GitHub user skunert opened a pull request: https://github.com/apache/incubator-flink/pull/39 Documentation update markdown I transformed java_api_quickstart, run_example_quickstart and setup_quickstart to markdown and updated the links. I gave up on doing everything in markdown for

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Stephan Ewen
@everyone interested in the Tez work: I created a JIRA Issue with a brief summary of the current status and plans: https://issues.apache.org/jira/browse/FLINK-972 I was thinking about a brief dedicate Tez Hangout next week. Please post here, if you would like to have a Hangout on Flink&Tez next w

[GitHub] incubator-flink pull request: [FLINK-971] Configure PooledByteBufA...

2014-06-24 Thread uce
GitHub user uce opened a pull request: https://github.com/apache/incubator-flink/pull/38 [FLINK-971] Configure PooledByteBufAllocator in NettyConnectionManager ...instead of using the default allocator Configuration: - 0 heap arenas, - n direct arenas (where

[jira] [Created] (FLINK-972) Run Flink on Tez

2014-06-24 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-972: -- Summary: Run Flink on Tez Key: FLINK-972 URL: https://issues.apache.org/jira/browse/FLINK-972 Project: Flink Issue Type: New Feature Components: New Co

[GitHub] incubator-flink pull request: hadoopcompatibility: Implementations...

2014-06-24 Thread atsikiridis
GitHub user atsikiridis opened a pull request: https://github.com/apache/incubator-flink/pull/37 hadoopcompatibility: Implementations of basic programming interfaces and... ... a basic driver. * wrappers for Mapper, Reducer and Combiner (as a local Reducer) on the new Java

[jira] [Created] (FLINK-971) Configure Netty ByteBufAllocator instead of using the default configuration

2014-06-24 Thread Ufuk Celebi (JIRA)
Ufuk Celebi created FLINK-971: - Summary: Configure Netty ByteBufAllocator instead of using the default configuration Key: FLINK-971 URL: https://issues.apache.org/jira/browse/FLINK-971 Project: Flink

Re: Java Primitive Collections in Flink

2014-06-24 Thread Stephan Ewen
I agree with Sean. Fastutil does all sorts of combinations of primitive key and value types, blowing the number of classes up. If you only need a simple growing array of primitive longs, it may be simpler to just implement it yourself.

Re: Some ideas for long-term Flink-related research and implementation projects

2014-06-24 Thread Kostas Tzoumas
Henry, I am currently travelling and be able to write more about this next week. The idea is to use Tez as the distributed engine, and port Flink's runtime operators (for joins, aggregation) etc on top of that. The Flink APIs and optimizer should not need many changes. This should be in theory pos

Re: [jira] [Created] (FLINK-970) Implement a first(n) operator

2014-06-24 Thread Ufuk Celebi
+1 Spark also has the more general `take(n)` operation, of which first() is a special case. it might make sense to go for `take(n)` and just have first() as a special case. On 24 Jun 2014, at 10:37, Timo Walther (JIRA) wrote: > Timo Walther created FLINK-970: > ---

Re: Starter Assignment

2014-06-24 Thread Fabian Hueske
Hi Sebastian, welcome to the Flink mailing list! Here's a list of tasks that should be good for getting in touch with Flink. They are mostly on the API or example level. https://issues.apache.org/jira/browse/FLINK-925 https://issues.apache.org/jira/browse/FLINK-873 https://issues.apache.org/jira

[jira] [Created] (FLINK-970) Implement a first(n) operator

2014-06-24 Thread Timo Walther (JIRA)
Timo Walther created FLINK-970: -- Summary: Implement a first(n) operator Key: FLINK-970 URL: https://issues.apache.org/jira/browse/FLINK-970 Project: Flink Issue Type: New Feature Rep

Re: Java Primitive Collections in Flink

2014-06-24 Thread Sean Owen
It's own implementation of a primitive map. The needs were limited, and two Scala classes did the trick. On Tue, Jun 24, 2014 at 9:18 AM, Robert Metzger wrote: > What did Spark use instead of fastutil? > > > On Tue, Jun 24, 2014 at 9:40 AM, Sean Owen wrote: > >> A word of caution -- fastutil is

Re: Java Primitive Collections in Flink

2014-06-24 Thread Robert Metzger
What did Spark use instead of fastutil? On Tue, Jun 24, 2014 at 9:40 AM, Sean Owen wrote: > A word of caution -- fastutil is a massive library, 20MB or so and 10K > files if I recall correctly. It was pulled out of Spark just because > it was making the deployment jars huge (and wasn't used muc

Re: Java Primitive Collections in Flink

2014-06-24 Thread Sean Owen
A word of caution -- fastutil is a massive library, 20MB or so and 10K files if I recall correctly. It was pulled out of Spark just because it was making the deployment jars huge (and wasn't used much). Make sure it's worth it. On Tue, Jun 24, 2014 at 8:33 AM, Robert Waury wrote: > Okay, > > I'm

Re: Java Primitive Collections in Flink

2014-06-24 Thread Robert Waury
Okay, I'm going to add fastutil to the dependencies in my next pull request. Cheers, Robert On Jun 20, 2014 8:52 AM, "Sebastian Schelter" wrote: > +1 for fastutils > > > On 06/20/2014 08:50 AM, Robert Metzger wrote: > >> Hi Robert, >> >> The Apache Commons Primitives Collection project seems to