looks like a misssing class issue? what makes you think its serialization? shapeless does indeed have a lot of helper classes that get sucked in and are not serializable. see here: https://groups.google.com/forum/#!topic/shapeless-dev/05_DXnoVnI4
and for a project that uses shapeless in spark see here: https://github.com/tresata/spark-columnar On Wed, Oct 29, 2014 at 7:05 PM, Simon Hafner <reactorm...@gmail.com> wrote: > I tried using shapeless HLists as data storage for data inside spark. > Unsurprisingly, it failed. The deserialization isn't well-defined because > of > all the implicits used by shapeless. How could I make it work? > > Sample Code: > > /* SimpleApp.scala */ > import org.apache.spark.SparkContext > import org.apache.spark.SparkContext._ > import org.apache.spark.SparkConf > import shapeless._ > import ops.hlist._ > > object SimpleApp { > def main(args: Array[String]) { > val logFile = "/tmp/README.md" // Should be some file on your system > val conf = new SparkConf().setAppName("Simple Application") > val sc = new SparkContext(conf) > val logData = sc.textFile(logFile, 2).cache() > val numAs = logData > .map(line => line :: HNil) > .filter(_.select[String].contains("a")) > .count() > println("Lines with a: %s".format(numAs)) > } > } > > Error: > > Exception in thread "main" java.lang.NoClassDefFoundError: > shapeless/$colon$colon > at SimpleApp$.main(SimpleApp.scala:15) > at SimpleApp.main(SimpleApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
