I saw another report so I filed it already: Filed as: https://issues.apache.org/jira/browse/SPARK-8470
On Thu, Jun 18, 2015 at 4:07 PM, Chad Urso McDaniel <cha...@gmail.com> wrote: > We're using the normal command line: > --- > bin/spark-submit --properties-file ./spark-submit.conf --class > com.rr.data.visits.VisitSequencerRunner > ./mvt-master-SNAPSHOT-jar-with-dependencies.jar > --- > > Our jar contains both com.rr.data.visits.orc.OrcReadWrite (which you can > see in the stack trace) and the unfound com.rr.data.Visit. > > I'll open a Jira ticket > > > On Thu, Jun 18, 2015 at 3:26 PM Michael Armbrust <mich...@databricks.com> > wrote: > >> How are you adding com.rr.data.Visit to spark? With --jars? It is >> possible we are using the wrong classloader. Could you open a JIRA? >> >> On Thu, Jun 18, 2015 at 2:56 PM, Chad Urso McDaniel <cha...@gmail.com> >> wrote: >> >>> We are seeing class exceptions when converting to a DataFrame. >>> Anyone out there with some suggestions on what is going on? >>> >>> Our original intention was to use a HiveContext to write ORC and we say >>> the error there and have narrowed it down. >>> >>> This is an example of our code: >>> --- >>> def saveVisitsAsOrcFile(sqlContext: SQLContext, rdd: RDD[Visit], >>> outputDir: String) { >>> // works!: println("rdd count: " + rdd.map(_.clicks.size).sum) >>> >>> import sqlContext.implicits._ >>> // scala.ScalaReflectionException: class com.rr.data.Visit >>> print("rdd.toDF.count:" + rdd >>> .toDF() >>> .count()) >>> --- >>> This runs locally, but when using spark-submit with 1.4 we get: >>> >>> >>> Exception in thread "main" scala.ScalaReflectionException: class >>> com.rr.data.Visit in JavaMirror with >>> sun.misc.Launcher$AppClassLoader@5c647e05 of type class >>> sun.misc.Launcher$AppClassLoader with classpath >>> [file:/home/candiru/tewfik/,file:/home/candiru/tewfik/spark-1.4.0-bin-tewfik-spark/conf/,file:/home/candiru/tewfik/spark-1.4.0-bin-tewfik-spark/lib/spark-assembly-1.4.0-hadoop2.0.0-mr1-cdh4.2.0.jar,file:/home/candiru/tewfik/spark-1.4.0-bin-tewfik-spark/lib/datanucleus-api-jdo-3.2.6.jar,file:/home/candiru/tewfik/spark-1.4.0-bin-tewfik-spark/lib/datanucleus-core-3.2.10.jar,file:/home/candiru/tewfik/spark-1.4.0-bin-tewfik-spark/lib/datanucleus-rdbms-3.2.9.jar] >>> and parent being sun.misc.Launcher$ExtClassLoader@1c79d093 of type >>> class sun.misc.Launcher$ExtClassLoader with classpath >>> [file:/usr/java/jdk1.8.0_05/jre/lib/ext/cldrdata.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/dnsns.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/jfxrt.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/localedata.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/nashorn.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/sunec.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/sunjce_provider.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/sunpkcs11.jar,file:/usr/java/jdk1.8.0_05/jre/lib/ext/zipfs.jar] >>> and parent being primordial classloader with boot classpath >>> [/usr/java/jdk1.8.0_05/jre/lib/resources.jar:/usr/java/jdk1.8.0_05/jre/lib/rt.jar:/usr/java/jdk1.8.0_05/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_05/jre/lib/jsse.jar:/usr/java/jdk1.8.0_05/jre/lib/jce.jar:/usr/java/jdk1.8.0_05/jre/lib/charsets.jar:/usr/java/jdk1.8.0_05/jre/lib/jfr.jar:/usr/java/jdk1.8.0_05/jre/classes] >>> not found. >>> at >>> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123) >>> at >>> scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22) >>> at >>> com.rr.data.visits.orc.OrcReadWrite$$typecreator2$1.apply(OrcReadWrite.scala:36) >>> at >>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) >>> at >>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) >>> at >>> org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:71) >>> at >>> org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:59) >>> at >>> org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:28) >>> at >>> org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:410) >>> at >>> org.apache.spark.sql.SQLContext$implicits$.rddToDataFrameHolder(SQLContext.scala:335) >>> at >>> com.rr.data.visits.orc.OrcReadWrite$.saveVisitsAsOrcFile(OrcReadWrite.scala:36) >>> at >>> com.rr.data.visits.VisitSequencerRunner$.main(VisitSequencerRunner.scala:43) >>> at >>> com.rr.data.visits.VisitSequencerRunner.main(VisitSequencerRunner.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:483) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) >>> at >>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) >>> at >>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> >>