Spark 1.4 RDD to DF fails with toDF()

[Gheorghe Postelnicu]

Hi,

The following code fails when compiled from SBT:

package main.scala

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext

object TestMain {
  def main(args: Array[String]): Unit = {
    implicit val sparkContext = new SparkContext()
    val sqlContext = new SQLContext(sparkContext)
    import sqlContext.implicits._
    sparkContext.parallelize(1 to 10).map(i => (i,
i.toString)).toDF("intCol", "strCol")
  }
}

with the following error:

15/09/07 21:39:21 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NoSuchMethodError:
scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
at main.scala.Bof$.main(Bof.scala:14)
at main.scala.Bof.main(Bof.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/09/07 21:39:22 INFO SparkContext: Invoking stop() from shutdown hook

whereas the code above works in a spark shell.

The code is compiled using Scala 2.11.6 and precompiled Spark 1.4.1

Any suggestion on how to fix this would be much appreciated.

Best,
Gheorghe