You are mixing the 1.0.0 Spark SQL jar with Spark 1.4.0 jars in your build file
Sent from my rotary phone.
On Jul 14, 2015, at 7:57 AM, ashwang168 ashw...@mit.edu wrote:
Hello!
I am currently using Spark 1.4.0, scala 2.10.4, and sbt 0.13.8 to try and
create a jar file from a scala file (attached above) and run it using
spark-submit. I am also using Hive, Hadoop 2.6.0-cdh5.4.0 which has the
files that I'm trying to read in.
Currently I am very confused about how the imports work and if there . I am
getting the error:
[error] bad symbolic reference. A signature in SQLContext.class refers to
term package
[error] in package org.apache.spark.sql which is not available.
[error] It may be completely missing from the current classpath, or the
version on
[error] the classpath might be incompatible with the version used when
compiling SQLContext.class.
[error] bad symbolic reference. A signature in SQLContext.class refers to
type Logging
[error] in value org.apache.spark.sql.package which is not available.
[error] It may be completely missing from the current classpath, or the
version on
[error] the classpath might be incompatible with the version used when
compiling SQLContext.class.
[error] bad symbolic reference. A signature in SchemaRDD.class refers to
term package
[error] in package org.apache.spark.sql which is not available.
[error] It may be completely missing from the current classpath, or the
version on
[error] the classpath might be incompatible with the version used when
compiling SchemaRDD.class.
[error] /root/awang/time/rddSpark/create/src/main/scala/create.scala:20:
value implicits is not a member of org.apache.spark.sql.SQLContext
...
[error] /root/awang/time/rddSpark/create/src/main/scala/create.scala:39:
value toDF is not a member of org.apache.spark.rdd.RDD[TSTData]
The imports in my code are:
import org.apache.spark._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql
import org.apache.spark.sql._
and in the object Create :
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._
import sqlContext.implicits._
import sqlContext.createSchemaRDD
My libraryDependencies are:
libraryDependencies ++= Seq(
// spark will already be on classpath when using spark-submit.
// marked as provided, so that it isn't included in assembly.
org.apache.spark %% spark-catalyst % 1.4.0 % provided,
org.apache.spark %% spark-sql % 1.0.0)
so why is package org.apache.spark.sql not available?
Also, what are the correct imports to get this working?
I'm using sbt assembly to try to compile these files, and would really
appreciate any help.
Thanks,
Ashley Wang
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/correct-Scala-Imports-for-creating-DFs-from-RDDs-tp23829.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
