Re: Dataset Question: No Encoder found for Set[(scala.Long, scala.Long)]

Hi Koert, Thank you for your help! GOT IT! Best Regards, Jerry On Wed, Feb 1, 2017 at 6:24 PM, Koert Kuipers wrote: > you can still use it as Dataset[Set[X]]. all transformations should work > correctly. > > however dataset.schema will show binary type, and dataset.show

Re: Dataset Question: No Encoder found for Set[(scala.Long, scala.Long)]

you can still use it as Dataset[Set[X]]. all transformations should work correctly. however dataset.schema will show binary type, and dataset.show will show bytes (unfortunately). for example: scala> implicit def setEncoder[X]: Encoder[Set[X]] = Encoders.kryo[Set[X]] setEncoder: [X]=>

Re: Dataset Question: No Encoder found for Set[(scala.Long, scala.Long)]

Hi Koert, Thanks for the tips. I tried to do that but the column's type is now Binary. Do I get the Set[X] back in the Dataset? Best Regards, Jerry On Tue, Jan 31, 2017 at 8:04 PM, Koert Kuipers wrote: > set is currently not supported. you can use kryo encoder. there is

Re: Dataset Question: No Encoder found for Set[(scala.Long, scala.Long)]

set is currently not supported. you can use kryo encoder. there is no other work around that i know of. import org.apache.spark.sql.{ Encoder, Encoders } implicit def setEncoder[X]: Encoder[Set[X]] = Encoders.kryo[Set[X]] On Tue, Jan 31, 2017 at 7:33 PM, Jerry Lam wrote:

Dataset Question: No Encoder found for Set[(scala.Long, scala.Long)]

Hi guys, I got an exception like the following, when I tried to implement a user defined aggregation function. Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for Set[(scala.Long, scala.Long)] The Set[(Long, Long)] is a field in the case class which is the