Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-10 Thread kant kodali
I have also tried these. And none of them actually compile. dataset.map(new MapFunction>>() { @Override public Seq> call(String input) throws Exception { List> temp = new ArrayList<>(); temp.add(new

Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-09 Thread kant kodali
Tried the following. dataset.map(new MapFunction>>() { @Override public List> call(String input) throws Exception { List> temp = new ArrayList<>(); temp.add(new HashMap());

Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-09 Thread Koert Kuipers
if you are willing to use kryo encoder you can do your original Dataset< List Seq(1,2,3).toDS.map(x => if (x % 2 == 0) x else x.toString)(org.apache.spark.sql.Encoders.kryo[Any]).map{ (x: Any)

Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-09 Thread kant kodali
Hi Koert, Thanks! If I have this Dataset>> what would be the Enconding?is it Encoding.kryo(Seq.class) ? Also shouldn't List be supported? Should I create a ticket for this? On Mon, Oct 9, 2017 at 6:10 AM, Koert Kuipers wrote: > it supports

Re: Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-09 Thread Koert Kuipers
it supports Dataset>> where X must be a supported type also. Object is not a supported type. On Mon, Oct 9, 2017 at 7:36 AM, kant kodali wrote: > Hi All, > > I am wondering if spark supports Dataset>> ? > > when I do the following

Does Spark 2.2.0 support Dataset<List<Map<String,Object>>> ?

2017-10-09 Thread kant kodali
Hi All, I am wondering if spark supports Dataset>> ? when I do the following it says no map function available? Dataset>> resultDs = ds.map(lambda, Encoders.bean(List.class)); Thanks!