[jira] [Created] (FLINK-3855) Upgrade Jackson version

2016-04-29 Thread Tatu Saloranta (JIRA)
Tatu Saloranta created FLINK-3855: - Summary: Upgrade Jackson version Key: FLINK-3855 URL: https://issues.apache.org/jira/browse/FLINK-3855 Project: Flink Issue Type: Improvement Com

[jira] [Created] (FLINK-3854) Support Avro key-value rolling sink

2016-04-29 Thread Igor Berman (JIRA)
Igor Berman created FLINK-3854: -- Summary: Support Avro key-value rolling sink Key: FLINK-3854 URL: https://issues.apache.org/jira/browse/FLINK-3854 Project: Flink Issue Type: Improvement

[VOTE] Release Apache Flink 1.0.3 (RC2)

2016-04-29 Thread Ufuk Celebi
Dear Flink community, Please vote on releasing the following candidate as Apache Flink version 1.0.3. The commit to be voted on: 51daef0063169ee276cc4877d52ea42731d884c1 Branch: release-1.0.3-rc2 (see https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-1.

[jira] [Created] (FLINK-3853) Reduce object creation in Gelly utility mappers

2016-04-29 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-3853: - Summary: Reduce object creation in Gelly utility mappers Key: FLINK-3853 URL: https://issues.apache.org/jira/browse/FLINK-3853 Project: Flink Issue Type: Improveme

Re: groupBy on a Dataset of Maps

2016-04-29 Thread Punit Naik
I forgot to mention that in this code my out.collect method is outputting a tuple of Map[key,value] and the count as Int. On Fri, Apr 29, 2016 at 4:53 PM, Punit Naik wrote: > Anyways, I fixed it. To you groupBy you should attach this: > > .reduceGroup { > (in, out: org.apache.flink.util.Co

Re: groupBy on a Dataset of Maps

2016-04-29 Thread Punit Naik
Anyways, I fixed it. To you groupBy you should attach this: .reduceGroup { (in, out: org.apache.flink.util.Collector[(Map[key,value], Int)]) => var v:Int = 0; var k:Map[key,value]=Map() for (t <- in) { v+=1; k=t } out.collect((k,v))

TypeVariable problems

2016-04-29 Thread Martin Neumann
Hej, I have a construct of different generic classes stacked on each other to create a library (so the type variables get handed on). And I have some trouble getting it to work. The current offender is a Class with 3 type variables internally it calls: .fold(new Tuple3<>(keyInit ,new Tuple2(0d,0

Re: groupBy on a Dataset of Maps

2016-04-29 Thread Punit Naik
What if after grouping I wanted to count the occurrences of the key "ga_date"? On Fri, Apr 29, 2016 at 2:15 PM, Stefano Baghino < stefano.bagh...@radicalbit.io> wrote: > The `get` method on the Scala map returns an Option, which is not > (currently) a valid key type for Flink (but there's ongoing

[RESULT] [VOTE] Release Apache Flink 1.0.3 (RC1)

2016-04-29 Thread Ufuk Celebi
This vote is cancelled in favour of RC2. On Fri, Apr 29, 2016 at 10:10 AM, Ufuk Celebi wrote: > Dear Flink community, > > Please vote on releasing the following candidate as Apache Flink version > 1.0.3. > > The commit to be voted on: > 06c5d7651dd2ef670d94ba0d6f0bb922a0aafd1b > > Branch: > rele

Re: RichMapPartitionFunction - problems with collect

2016-04-29 Thread Till Rohrmann
You have to persist it and read from it for the subsequent operations. Take a look at the FlinkMLTools.persist methods. Cheers, Till ​ On Thu, Apr 28, 2016 at 6:14 PM, Sergio Ramírez wrote: > Hello, > > OK, now I understand everything. So if I want to re-use my DataSet in > several different op

Re: [VOTE] Release Apache Flink 1.0.3 (RC1)

2016-04-29 Thread Ufuk Celebi
-1 Just had a discussion with Till about 014a686 - [FLINK-3800] [jobmanager] Terminate ExecutionGraphs properly We think that we have to revert this for 1.0, because it needs some follow-up work in order to ensure that jobs are not removed unexpectedly. – Ufuk On Fri, Apr 29, 2016 at 10:10 AM

Re: groupBy on a Dataset of Maps

2016-04-29 Thread Punit Naik
okay I'll take all your points into consideration. On Fri, Apr 29, 2016 at 2:15 PM, Stefano Baghino < stefano.bagh...@radicalbit.io> wrote: > The `get` method on the Scala map returns an Option, which is not > (currently) a valid key type for Flink (but there's ongoing work on this > [1]). Flink

[jira] [Created] (FLINK-3852) Use a StreamExecutionEnvironment in the quickstart job skeleton

2016-04-29 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-3852: - Summary: Use a StreamExecutionEnvironment in the quickstart job skeleton Key: FLINK-3852 URL: https://issues.apache.org/jira/browse/FLINK-3852 Project: Flink

Re: groupBy on a Dataset of Maps

2016-04-29 Thread Stefano Baghino
The `get` method on the Scala map returns an Option, which is not (currently) a valid key type for Flink (but there's ongoing work on this [1]). Flink must be aware of how to use a particular type as a key if you want to group by a value of said type. See the advanced DataSet concepts in the offici

groupBy on a Dataset of Maps

2016-04-29 Thread Punit Naik
Below is my code: val env = ExecutionEnvironment.getExecutionEnvironment val data=env.readTextFile("file:///home/punit/test").flatMap( line => JSON.parseFull(line) ) val j=data.flatMap{ _ match {case map: Map[String, Any] => {List(Map("ga_date" -> map.get("ga_dateHour")

[VOTE] Release Apache Flink 1.0.3 (RC1)

2016-04-29 Thread Ufuk Celebi
Dear Flink community, Please vote on releasing the following candidate as Apache Flink version 1.0.3. The commit to be voted on: 06c5d7651dd2ef670d94ba0d6f0bb922a0aafd1b Branch: release-1.0.3-rc1 (see https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-1.

Re: Read JSON file as input

2016-04-29 Thread Stefano Baghino
Great, Punit, I'm glad I've been of some help. If you have similar issues, feel free to write to the user mailing list, I believe you'll find help more easily there as you approach Flink. Happy hacking! :) On Thu, Apr 28, 2016 at 9:10 PM, Punit Naik wrote: > I am so sorry. Please ignore my previ