[ https://issues.apache.org/jira/browse/SPARK-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572217#comment-15572217 ]
Khalid Reid edited comment on SPARK-17890 at 10/13/16 3:17 PM: --------------------------------------------------------------- Hi Sean, I've created a small project [here|https://github.com/khalidr/Spark_17890] to reproduce the error using spark-submit. I noticed that things work fine when using an RDD but fails with the error below when I use the toDF() function. {noformat} Exception in thread "main" scala.ScalaReflectionException: class Foo not found. at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123) at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22) at Main$$typecreator3$1.apply(Main.scala:20) at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) at org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125) at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49) at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125) at Main$.delayedEndpoint$Main$1(Main.scala:20) at Main$delayedInit$body.apply(Main.scala:4) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at Main$.main(Main.scala:4) at Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/10/13 10:56:15 INFO SparkContext: Invoking stop() from shutdown hook {noformat} was (Author: kor): Hi Sean, I've created a small project to reproduce the error using spark-submit. I noticed that things work fine when using an RDD but fails when I use the toDF() function. https://github.com/khalidr/Spark_17890 > scala.ScalaReflectionException > ------------------------------ > > Key: SPARK-17890 > URL: https://issues.apache.org/jira/browse/SPARK-17890 > Project: Spark > Issue Type: Bug > Affects Versions: 2.0.1 > Environment: x86_64 GNU/Linux > Java(TM) SE Runtime Environment (build 1.8.0_60-b27) > Reporter: Khalid Reid > Priority: Minor > Labels: newbie > > Hello, > I am seeing an error message in spark-shell when I map a DataFrame to a > Seq\[Foo\]. However, things work fine when I use flatMap. > {noformat} > scala> case class Foo(value:String) > defined class Foo > scala> val df = sc.parallelize(List(1,2,3)).toDF > df: org.apache.spark.sql.DataFrame = [value: int] > scala> df.map{x => Seq.empty[Foo]} > scala.ScalaReflectionException: object $line14.$read not found. > at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162) > at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22) > at $typecreator1$1.apply(<console>:29) > at > scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) > at > org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125) > at > scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) > at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) > at > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49) > at > org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125) > ... 48 elided > scala> df.flatMap{_ => Seq.empty[Foo]} //flatMap works > res2: org.apache.spark.sql.Dataset[Foo] = [value: string] > {noformat} > I am seeing the same error reported > [here|https://issues.apache.org/jira/browse/SPARK-8465?jql=text%20~%20%22scala.ScalaReflectionException%22] > when I use spark-submit. > I am new to Spark but I don't expect this to throw an exception. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org