Thanks Jacek, I thought there was some dependency issue. This did the trick
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" *libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" % "provided"* I use a shell script that builds the jar file depending on type (sbt, mvn, assembly) and submits it via spark-submit .. ./local.ksh -A ETL_scratchpad_dummy -T sbt As I understand "provided" means that the dependencies will be provided at run-time (spark-submit) through the jar files but they are not needed at compile time. Having said that am I correct that error message below [error] bad symbolic reference. A signature in HiveContext.class refers to type Logging [error] in package org.apache.spark which is not available. [error] It may be completely missing from the current classpath, or the version on [error] the classpath might be incompatible with the version used when compiling HiveContext.class. [error] one error found [error] (compile:compileIncremental) Compilation failed meant that some form of libraries incompatibility was happening at compile time? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 14 August 2016 at 19:11, Jacek Laskowski <ja...@japila.pl> wrote: > Go to spark-shell and do :imports. You'll see all the imports and you > could copy and paste them in your app. (but there are not many > honestly and that won't help you much) > > HiveContext lives in spark-hive. You don't need spark-sql and > spark-hive since the latter uses the former as a dependency (unless > you're using types that come from the other dependencies). You don't > need spark-core either. Make the dependencies simpler by: > > libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" > > and mark it % Provided. > > The reason for provided is that you don't need that for uber-jar that > you're going to spark-submit. > > Don't forget to reload your session of sbt you're compiling in. Unsure > how you do it so quit your sbt session and do `sbt compile`. > > Ask away if you need more details. > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Sun, Aug 14, 2016 at 9:26 AM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > The issue is on Spark shell this works OK > > > > Spark context Web UI available at http://50.140.197.217:55555 > > Spark context available as 'sc' (master = local, app id = > > local-1471191662017). > > Spark session available as 'spark'. > > Welcome to > > ____ __ > > / __/__ ___ _____/ /__ > > _\ \/ _ \/ _ `/ __/ '_/ > > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0 > > /_/ > > Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java > > 1.8.0_77) > > Type in expressions to have them evaluated. > > Type :help for more information. > > scala> import org.apache.spark.SparkContext > > scala> import org.apache.spark.SparkConf > > scala> import org.apache.spark.sql.Row > > scala> import org.apache.spark.sql.hive.HiveContext > > scala> import org.apache.spark.sql.types._ > > scala> import org.apache.spark.sql.SparkSession > > scala> import org.apache.spark.sql.functions._ > > > > The code itself > > > > > > scala> val conf = new SparkConf(). > > | setAppName("ETL_scratchpad_dummy"). > > | set("spark.driver.allowMultipleContexts", "true"). > > | set("enableHiveSupport","true") > > conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@33215ffb > > > > scala> val sc = new SparkContext(conf) > > sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@ > 3cbfdf5c > > > > scala> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > > warning: there was one deprecation warning; re-run with -deprecation for > > details > > HiveContext: org.apache.spark.sql.hive.HiveContext = > > org.apache.spark.sql.hive.HiveContext@2152fde5 > > > > scala> HiveContext.sql("use oraclehadoop") > > res0: org.apache.spark.sql.DataFrame = [] > > > > I think I am getting something missing here a dependency > > > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd > OABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > The > > author will in no case be liable for any monetary damages arising from > such > > loss, damage or destruction. > > > > > > > > > > On 14 August 2016 at 17:16, Koert Kuipers <ko...@tresata.com> wrote: > >> > >> HiveContext is gone > >> > >> SparkSession now combines functionality of SqlContext and HiveContext > (if > >> hive support is available) > >> > >> On Sun, Aug 14, 2016 at 12:12 PM, Mich Talebzadeh > >> <mich.talebza...@gmail.com> wrote: > >>> > >>> Thanks Koert, > >>> > >>> I did that before as well. Anyway this is dependencies > >>> > >>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" > >>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" > >>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" > >>> > >>> > >>> and the error > >>> > >>> > >>> [info] Compiling 1 Scala source to > >>> /data6/hduser/scala/ETL_scratchpad_dummy/target/scala-2.10/classes... > >>> [error] > >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/ > scala/ETL_scratchpad_dummy.scala:4: > >>> object hive is not a member of package org.apache.spark.sql > >>> [error] import org.apache.spark.sql.hive.HiveContext > >>> [error] ^ > >>> [error] > >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/ > scala/ETL_scratchpad_dummy.scala:20: > >>> object hive is not a member of package org.apache.spark.sql > >>> [error] val HiveContext = new org.apache.spark.sql.hive. > HiveContext(sc) > >>> > >>> > >>> > >>> > >>> > >>> Dr Mich Talebzadeh > >>> > >>> > >>> > >>> LinkedIn > >>> https://www.linkedin.com/profile/view?id= > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > >>> > >>> > >>> http://talebzadehmich.wordpress.com > >>> > >>> > >>> Disclaimer: Use it at your own risk. Any and all responsibility for any > >>> loss, damage or destruction of data or any other property which may > arise > >>> from relying on this email's technical content is explicitly > disclaimed. The > >>> author will in no case be liable for any monetary damages arising from > such > >>> loss, damage or destruction. > >>> > >>> > >>> > >>> > >>> On 14 August 2016 at 17:00, Koert Kuipers <ko...@tresata.com> wrote: > >>>> > >>>> you cannot mix spark 1 and spark 2 jars > >>>> > >>>> change this > >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" > >>>> to > >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" > >>>> > >>>> On Sun, Aug 14, 2016 at 11:58 AM, Mich Talebzadeh > >>>> <mich.talebza...@gmail.com> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> In Spark 2 I am using sbt or mvn to compile my scala program. This > used > >>>>> to compile and run perfectly with Spark 1.6.1 but now it is throwing > error > >>>>> > >>>>> > >>>>> I believe the problem is here. I have > >>>>> > >>>>> name := "scala" > >>>>> version := "1.0" > >>>>> scalaVersion := "2.11.7" > >>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" > >>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" > >>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" > >>>>> > >>>>> However the error I am getting is > >>>>> > >>>>> [error] bad symbolic reference. A signature in HiveContext.class > refers > >>>>> to type Logging > >>>>> [error] in package org.apache.spark which is not available. > >>>>> [error] It may be completely missing from the current classpath, or > the > >>>>> version on > >>>>> [error] the classpath might be incompatible with the version used > when > >>>>> compiling HiveContext.class. > >>>>> [error] one error found > >>>>> [error] (compile:compileIncremental) Compilation failed > >>>>> > >>>>> > >>>>> And this is the code > >>>>> > >>>>> import org.apache.spark.SparkContext > >>>>> import org.apache.spark.SparkConf > >>>>> import org.apache.spark.sql.Row > >>>>> import org.apache.spark.sql.hive.HiveContext > >>>>> import org.apache.spark.sql.types._ > >>>>> import org.apache.spark.sql.SparkSession > >>>>> import org.apache.spark.sql.functions._ > >>>>> object ETL_scratchpad_dummy { > >>>>> def main(args: Array[String]) { > >>>>> val conf = new SparkConf(). > >>>>> setAppName("ETL_scratchpad_dummy"). > >>>>> set("spark.driver.allowMultipleContexts", "true"). > >>>>> set("enableHiveSupport","true") > >>>>> val sc = new SparkContext(conf) > >>>>> //import sqlContext.implicits._ > >>>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > >>>>> HiveContext.sql("use oraclehadoop") > >>>>> > >>>>> > >>>>> Anyone has come across this > >>>>> > >>>>> > >>>>> > >>>>> Dr Mich Talebzadeh > >>>>> > >>>>> > >>>>> > >>>>> LinkedIn > >>>>> https://www.linkedin.com/profile/view?id= > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>>>> > >>>>> > >>>>> > >>>>> http://talebzadehmich.wordpress.com > >>>>> > >>>>> > >>>>> Disclaimer: Use it at your own risk. Any and all responsibility for > any > >>>>> loss, damage or destruction of data or any other property which may > arise > >>>>> from relying on this email's technical content is explicitly > disclaimed. The > >>>>> author will in no case be liable for any monetary damages arising > from such > >>>>> loss, damage or destruction. > >>>>> > >>>>> > >>>> > >>>> > >>> > >> > > >
