I have some code that was producing OOM during shuffle and was RDD. So, upon direction by a member of Databricks I started covering to Datasets. However, when we did we are getting an error that seems to be not liking something within one of our case classes.
Peter Halliday [2016-06-08 19:12:22,083] ERROR org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class threw exception: java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation] - field (class: "scala.collection.immutable.Set", name: "violations") - root class: "com.here.probe.ingestion.converters.ProbeValidation" java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation] - field (class: "scala.collection.immutable.Set", name: "violations") - root class: "com.here.probe.ingestion.converters.ProbeValidation" at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594) at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494) at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490) at org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54) at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41) at com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116) at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86) at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53) at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542) [2016-06-08 19:12:22,086] INFO org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation] - field (class: "scala.collection.immutable.Set", name: "violations") - root class: "com.here.probe.ingestion.converters.ProbeValidation”)