Re: Generic Dataset Aggregator

2016-01-26 Thread Arkadiusz Bicz
Hi Deenar, You just need to encapsulate Array in Case Class ( you can not define case class inside spark shell as it can not be inner class) import com.hsbc.rsl.spark.aggregation.MinVectorAggFunction import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions.Aggregator

Generic Dataset Aggregator

2016-01-25 Thread Deenar Toraskar
Hi All https://docs.cloud.databricks.com/docs/spark/1.6/index.html#examples/Dataset%20Aggregator.html I have been converting my UDAFs to Dataset (Dataset's are cool BTW) Aggregators. I have an ArraySum aggregator that does an element wise sum or arrays. I have got the simple version working, but