I tried to convert an RDD to a data frame using the example codes on spark
website
case class Person(name: String, age: Int)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val people =
sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p
=> Person(p(0), p(1).trim.toInt)).toDF()
It compile fine using sbt. But in IntelliJ IDEA 14.0.3, it fail to compile and
generate the following error
Error:(289, 23) value implicits is not a member of
org.apache.spark.sql.SQLContext
import sqlContext.implicits._
^
What is the problem here? Is there any work around (e.g. do explicit
conversion)?
Here is my build.sbt
name := "my-project"
version := "0.2"
scalaVersion := "2.10.4"
val sparkVersion = "1.3.1"
val luceneVersion = "4.10.2"
libraryDependencies <<= scalaVersion {
scala_version => Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
"spark.jobserver" % "job-server-api" % "0.4.1" % "provided",
"org.scalatest" %% "scalatest" % "2.2.1" % "test"
)
}
resolvers += "Spark Packages Repo" at
"http://dl.bintray.com/spark-packages/maven"
resolvers += Resolver.mavenLocal
Ningjun