Hmm... look like the console command still starts a Spark 1.3.0 with Scala 2.11.6 even I changed them in build.sbt.
So the test with 1.2.1 is not valid. Jianshi On Fri, Mar 13, 2015 at 2:34 PM, Jianshi Huang <jianshi.hu...@gmail.com> wrote: > I've confirmed it only failed in console started by SBT. > > I'm using sbt-spark-package plugin, and the initialCommands look like this > (I added implicit sqlContext to it): > > > show console::initialCommands > [info] println("Welcome to\n" + > [info] " ____ __\n" + > [info] " / __/__ ___ _____/ /__\n" + > [info] " _\\ \\/ _ \\/ _ `/ __/ '_/\n" + > [info] " /___/ .__/\\_,_/_/ /_/\\_\\ version \"1.3.0-rc2\"\n" + > [info] " /_/\n" + > [info] "Using Scala \"2.11.6\"\n") > [info] > [info] import org.apache.spark.SparkContext._ > [info] > [info] val sc = { > [info] val conf = new org.apache.spark.SparkConf() > [info] .setMaster("local") > [info] .setAppName("Sbt console + Spark!") > [info] new org.apache.spark.SparkContext(conf) > [info] } > [info] println("Created spark context as sc.") > [info] > [info] def time[T](f: => T): T = { > [info] import System.{currentTimeMillis => now} > [info] val start = now > [info] try { f } finally { println("Elapsed: " + (now - start)/1000.0 + > " s") } > [info] } > [info] > [info] @transient val sqlc = new org.apache.spark.sql.SQLContext(sc) > [info] implicit def sqlContext = sqlc > [info] import sqlc._ > > > Jianshi > > > On Fri, Mar 13, 2015 at 3:10 AM, Jianshi Huang <jianshi.hu...@gmail.com> > wrote: > >> BTW, I was running tests from SBT when I get the errors. One test turn a >> Seq of case class to DataFrame. >> >> I also tried run similar code in the console, but failed with same error. >> >> I tested both Spark 1.3.0-rc2 and 1.2.1 with Scala 2.11.6 and 2.10.4 >> >> Any idea? >> >> Jianshi >> >> On Fri, Mar 13, 2015 at 2:23 AM, Jianshi Huang <jianshi.hu...@gmail.com> >> wrote: >> >>> Same issue here. But the classloader in my exception is somehow >>> different. >>> >>> scala.ScalaReflectionException: class >>> org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with >>> java.net.URLClassLoader@53298398 of type class java.net.URLClassLoader >>> with classpath >>> >>> >>> Jianshi >>> >>> On Sun, Mar 1, 2015 at 9:32 AM, Michael Armbrust <mich...@databricks.com >>> > wrote: >>> >>>> I think its possible that the problem is that the scala compiler is not >>>> being loaded by the primordial classloader (but instead by some child >>>> classloader) and thus the scala reflection mirror is failing to initialize >>>> when it can't find it. Unfortunately, the only solution that I know of is >>>> to load all required jars when the JVM starts. >>>> >>>> On Sat, Feb 28, 2015 at 5:26 PM, Ashish Nigam <ashnigamt...@gmail.com> >>>> wrote: >>>> >>>>> Also, can scala version play any role here? >>>>> I am using scala 2.11.5 but all spark packages have dependency to >>>>> scala 2.11.2 >>>>> Just wanted to make sure that scala version is not an issue here. >>>>> >>>>> On Sat, Feb 28, 2015 at 9:18 AM, Ashish Nigam <ashnigamt...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> I wrote a very simple program in scala to convert an existing RDD to >>>>>> SchemaRDD. >>>>>> But createSchemaRDD function is throwing exception >>>>>> >>>>>> Exception in thread "main" scala.ScalaReflectionException: class >>>>>> org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with >>>>>> primordial >>>>>> classloader with boot classpath [.....] not found >>>>>> >>>>>> >>>>>> Here's more info on the versions I am using - >>>>>> >>>>>> <scala.binary.version>2.11</scala.binary.version> >>>>>> <spark.version>1.2.1</spark.version> >>>>>> <scala.version>2.11.5</scala.version> >>>>>> >>>>>> Please let me know how can I resolve this problem. >>>>>> >>>>>> Thanks >>>>>> Ashish >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Jianshi Huang >>> >>> LinkedIn: jianshi >>> Twitter: @jshuang >>> Github & Blog: http://huangjs.github.com/ >>> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > > > -- > Jianshi Huang > > LinkedIn: jianshi > Twitter: @jshuang > Github & Blog: http://huangjs.github.com/ > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/