I have some code that was producing OOM during shuffle and was RDD. So, upon
direction by a member of Databricks I started covering to Datasets. However,
when we did we are getting an error that seems to be not liking something
within one of our case classes.
Peter Halliday
[2016-06-08 19:12:22,083] ERROR org.apache.spark.deploy.yarn.ApplicationMaster
[Driverhread] - User class threw exception:
java.lang.UnsupportedOperationException: No Encoder found for
Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation"
java.lang.UnsupportedOperationException: No Encoder found for
Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation"
at
org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594)
at
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494)
at
org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at
org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490)
at
org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
at
org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
at
com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116)
at
com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86)
at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53)
at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
[2016-06-08 19:12:22,086] INFO org.apache.spark.deploy.yarn.ApplicationMaster
[Driverhread] - Final app status: FAILED, exitCode: 15, (reason: User class
threw exception: java.lang.UnsupportedOperationException: No Encoder found for
Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation”)
