Hi, You should have all the deps being "provided" since they're provided by spark infra after you spark-submit the uber-jar for the app.
What's the "magic" in local.ksh? Why don't you sbt assembly and do spark-submit with the uber-jar? Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Aug 14, 2016 at 11:52 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Jacek, > > I thought there was some dependency issue. This did the trick > > libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" > libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" > libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" % > "provided" > > I use a shell script that builds the jar file depending on type (sbt, mvn, > assembly) and submits it via spark-submit .. > > ./local.ksh -A ETL_scratchpad_dummy -T sbt > > As I understand "provided" means that the dependencies will be provided at > run-time (spark-submit) through the jar files but they are not needed at > compile time. > > Having said that am I correct that error message below > > [error] bad symbolic reference. A signature in HiveContext.class refers to > type Logging > [error] in package org.apache.spark which is not available. > [error] It may be completely missing from the current classpath, or the > version on > [error] the classpath might be incompatible with the version used when > compiling HiveContext.class. > [error] one error found > [error] (compile:compileIncremental) Compilation failed > > meant that some form of libraries incompatibility was happening at compile > time? > > Cheers > > > Dr Mich Talebzadeh > > > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > http://talebzadehmich.wordpress.com > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > > > On 14 August 2016 at 19:11, Jacek Laskowski <ja...@japila.pl> wrote: >> >> Go to spark-shell and do :imports. You'll see all the imports and you >> could copy and paste them in your app. (but there are not many >> honestly and that won't help you much) >> >> HiveContext lives in spark-hive. You don't need spark-sql and >> spark-hive since the latter uses the former as a dependency (unless >> you're using types that come from the other dependencies). You don't >> need spark-core either. Make the dependencies simpler by: >> >> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" >> >> and mark it % Provided. >> >> The reason for provided is that you don't need that for uber-jar that >> you're going to spark-submit. >> >> Don't forget to reload your session of sbt you're compiling in. Unsure >> how you do it so quit your sbt session and do `sbt compile`. >> >> Ask away if you need more details. >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> >> >> On Sun, Aug 14, 2016 at 9:26 AM, Mich Talebzadeh >> <mich.talebza...@gmail.com> wrote: >> > The issue is on Spark shell this works OK >> > >> > Spark context Web UI available at http://50.140.197.217:55555 >> > Spark context available as 'sc' (master = local, app id = >> > local-1471191662017). >> > Spark session available as 'spark'. >> > Welcome to >> > ____ __ >> > / __/__ ___ _____/ /__ >> > _\ \/ _ \/ _ `/ __/ '_/ >> > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0 >> > /_/ >> > Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java >> > 1.8.0_77) >> > Type in expressions to have them evaluated. >> > Type :help for more information. >> > scala> import org.apache.spark.SparkContext >> > scala> import org.apache.spark.SparkConf >> > scala> import org.apache.spark.sql.Row >> > scala> import org.apache.spark.sql.hive.HiveContext >> > scala> import org.apache.spark.sql.types._ >> > scala> import org.apache.spark.sql.SparkSession >> > scala> import org.apache.spark.sql.functions._ >> > >> > The code itself >> > >> > >> > scala> val conf = new SparkConf(). >> > | setAppName("ETL_scratchpad_dummy"). >> > | set("spark.driver.allowMultipleContexts", "true"). >> > | set("enableHiveSupport","true") >> > conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@33215ffb >> > >> > scala> val sc = new SparkContext(conf) >> > sc: org.apache.spark.SparkContext = >> > org.apache.spark.SparkContext@3cbfdf5c >> > >> > scala> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >> > warning: there was one deprecation warning; re-run with -deprecation for >> > details >> > HiveContext: org.apache.spark.sql.hive.HiveContext = >> > org.apache.spark.sql.hive.HiveContext@2152fde5 >> > >> > scala> HiveContext.sql("use oraclehadoop") >> > res0: org.apache.spark.sql.DataFrame = [] >> > >> > I think I am getting something missing here a dependency >> > >> > >> > Dr Mich Talebzadeh >> > >> > >> > >> > LinkedIn >> > >> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > >> > >> > >> > http://talebzadehmich.wordpress.com >> > >> > >> > Disclaimer: Use it at your own risk. Any and all responsibility for any >> > loss, damage or destruction of data or any other property which may >> > arise >> > from relying on this email's technical content is explicitly disclaimed. >> > The >> > author will in no case be liable for any monetary damages arising from >> > such >> > loss, damage or destruction. >> > >> > >> > >> > >> > On 14 August 2016 at 17:16, Koert Kuipers <ko...@tresata.com> wrote: >> >> >> >> HiveContext is gone >> >> >> >> SparkSession now combines functionality of SqlContext and HiveContext >> >> (if >> >> hive support is available) >> >> >> >> On Sun, Aug 14, 2016 at 12:12 PM, Mich Talebzadeh >> >> <mich.talebza...@gmail.com> wrote: >> >>> >> >>> Thanks Koert, >> >>> >> >>> I did that before as well. Anyway this is dependencies >> >>> >> >>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" >> >>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" >> >>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" >> >>> >> >>> >> >>> and the error >> >>> >> >>> >> >>> [info] Compiling 1 Scala source to >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/target/scala-2.10/classes... >> >>> [error] >> >>> >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/scala/ETL_scratchpad_dummy.scala:4: >> >>> object hive is not a member of package org.apache.spark.sql >> >>> [error] import org.apache.spark.sql.hive.HiveContext >> >>> [error] ^ >> >>> [error] >> >>> >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/scala/ETL_scratchpad_dummy.scala:20: >> >>> object hive is not a member of package org.apache.spark.sql >> >>> [error] val HiveContext = new >> >>> org.apache.spark.sql.hive.HiveContext(sc) >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Dr Mich Talebzadeh >> >>> >> >>> >> >>> >> >>> LinkedIn >> >>> >> >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >>> >> >>> >> >>> >> >>> http://talebzadehmich.wordpress.com >> >>> >> >>> >> >>> Disclaimer: Use it at your own risk. Any and all responsibility for >> >>> any >> >>> loss, damage or destruction of data or any other property which may >> >>> arise >> >>> from relying on this email's technical content is explicitly >> >>> disclaimed. The >> >>> author will in no case be liable for any monetary damages arising from >> >>> such >> >>> loss, damage or destruction. >> >>> >> >>> >> >>> >> >>> >> >>> On 14 August 2016 at 17:00, Koert Kuipers <ko...@tresata.com> wrote: >> >>>> >> >>>> you cannot mix spark 1 and spark 2 jars >> >>>> >> >>>> change this >> >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >> >>>> to >> >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" >> >>>> >> >>>> On Sun, Aug 14, 2016 at 11:58 AM, Mich Talebzadeh >> >>>> <mich.talebza...@gmail.com> wrote: >> >>>>> >> >>>>> Hi, >> >>>>> >> >>>>> In Spark 2 I am using sbt or mvn to compile my scala program. This >> >>>>> used >> >>>>> to compile and run perfectly with Spark 1.6.1 but now it is throwing >> >>>>> error >> >>>>> >> >>>>> >> >>>>> I believe the problem is here. I have >> >>>>> >> >>>>> name := "scala" >> >>>>> version := "1.0" >> >>>>> scalaVersion := "2.11.7" >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >> >>>>> >> >>>>> However the error I am getting is >> >>>>> >> >>>>> [error] bad symbolic reference. A signature in HiveContext.class >> >>>>> refers >> >>>>> to type Logging >> >>>>> [error] in package org.apache.spark which is not available. >> >>>>> [error] It may be completely missing from the current classpath, or >> >>>>> the >> >>>>> version on >> >>>>> [error] the classpath might be incompatible with the version used >> >>>>> when >> >>>>> compiling HiveContext.class. >> >>>>> [error] one error found >> >>>>> [error] (compile:compileIncremental) Compilation failed >> >>>>> >> >>>>> >> >>>>> And this is the code >> >>>>> >> >>>>> import org.apache.spark.SparkContext >> >>>>> import org.apache.spark.SparkConf >> >>>>> import org.apache.spark.sql.Row >> >>>>> import org.apache.spark.sql.hive.HiveContext >> >>>>> import org.apache.spark.sql.types._ >> >>>>> import org.apache.spark.sql.SparkSession >> >>>>> import org.apache.spark.sql.functions._ >> >>>>> object ETL_scratchpad_dummy { >> >>>>> def main(args: Array[String]) { >> >>>>> val conf = new SparkConf(). >> >>>>> setAppName("ETL_scratchpad_dummy"). >> >>>>> set("spark.driver.allowMultipleContexts", "true"). >> >>>>> set("enableHiveSupport","true") >> >>>>> val sc = new SparkContext(conf) >> >>>>> //import sqlContext.implicits._ >> >>>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >> >>>>> HiveContext.sql("use oraclehadoop") >> >>>>> >> >>>>> >> >>>>> Anyone has come across this >> >>>>> >> >>>>> >> >>>>> >> >>>>> Dr Mich Talebzadeh >> >>>>> >> >>>>> >> >>>>> >> >>>>> LinkedIn >> >>>>> >> >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >>>>> >> >>>>> >> >>>>> >> >>>>> http://talebzadehmich.wordpress.com >> >>>>> >> >>>>> >> >>>>> Disclaimer: Use it at your own risk. Any and all responsibility for >> >>>>> any >> >>>>> loss, damage or destruction of data or any other property which may >> >>>>> arise >> >>>>> from relying on this email's technical content is explicitly >> >>>>> disclaimed. The >> >>>>> author will in no case be liable for any monetary damages arising >> >>>>> from such >> >>>>> loss, damage or destruction. >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >> >> > > > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
