Re: How to call a custom function from GroupByKey which takes Iterable[Row] as input and returns a Map[Int,String] as output in scala

Hi Vishal, Thanks for the solution. I was able to get it working for my scenario. Regarding the Task not serializable error, I still get it when I declare a function outside the main method. However, if I declare it inside the main "val func = {}", it is working fine for me. In case you have any

Re: How to call a custom function from GroupByKey which takes Iterable[Row] as input and returns a Map[Int,String] as output in scala

It seems Spark is not able to serialize your function code to worker nodes. I have tried to put a solution in simple set of commands. Maybe you can combine last four line into function. val arr = Array((1,"A","<20","0"), (1,"A",">20 & <40","1"), (1,"B",">20 & <40","0"), (1,"C",">20 & <40","0"),

How to call a custom function from GroupByKey which takes Iterable[Row] as input and returns a Map[Int,String] as output in scala

Hi, I have a scenario wherein my dataset has around 30 columns. It is basically user activity information. I need to group the information by each user and then for each column/activity parameter I need to find the percentage affinity for each value in that column for that user. Below is the

How to call a custom function from GroupByKey which takes Iterable[Row] as input and returns a Map[Int,String] as output in scala

Hi, I have a scenario wherein my dataset has around 30 columns. It is basically user activity information. I need to group the information by each user and then for each column/activity parameter I need to find the percentage affinity for each value in that column for that user. Below is the