Go to spark-shell and do :imports. You'll see all the imports and you could copy and paste them in your app. (but there are not many honestly and that won't help you much)
HiveContext lives in spark-hive. You don't need spark-sql and spark-hive since the latter uses the former as a dependency (unless you're using types that come from the other dependencies). You don't need spark-core either. Make the dependencies simpler by: libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" and mark it % Provided. The reason for provided is that you don't need that for uber-jar that you're going to spark-submit. Don't forget to reload your session of sbt you're compiling in. Unsure how you do it so quit your sbt session and do `sbt compile`. Ask away if you need more details. Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Aug 14, 2016 at 9:26 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > The issue is on Spark shell this works OK > > Spark context Web UI available at http://50.140.197.217:55555 > Spark context available as 'sc' (master = local, app id = > local-1471191662017). > Spark session available as 'spark'. > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0 > /_/ > Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.8.0_77) > Type in expressions to have them evaluated. > Type :help for more information. > scala> import org.apache.spark.SparkContext > scala> import org.apache.spark.SparkConf > scala> import org.apache.spark.sql.Row > scala> import org.apache.spark.sql.hive.HiveContext > scala> import org.apache.spark.sql.types._ > scala> import org.apache.spark.sql.SparkSession > scala> import org.apache.spark.sql.functions._ > > The code itself > > > scala> val conf = new SparkConf(). > | setAppName("ETL_scratchpad_dummy"). > | set("spark.driver.allowMultipleContexts", "true"). > | set("enableHiveSupport","true") > conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@33215ffb > > scala> val sc = new SparkContext(conf) > sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@3cbfdf5c > > scala> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) > warning: there was one deprecation warning; re-run with -deprecation for > details > HiveContext: org.apache.spark.sql.hive.HiveContext = > org.apache.spark.sql.hive.HiveContext@2152fde5 > > scala> HiveContext.sql("use oraclehadoop") > res0: org.apache.spark.sql.DataFrame = [] > > I think I am getting something missing here a dependency > > > Dr Mich Talebzadeh > > > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > http://talebzadehmich.wordpress.com > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > > > On 14 August 2016 at 17:16, Koert Kuipers <ko...@tresata.com> wrote: >> >> HiveContext is gone >> >> SparkSession now combines functionality of SqlContext and HiveContext (if >> hive support is available) >> >> On Sun, Aug 14, 2016 at 12:12 PM, Mich Talebzadeh >> <mich.talebza...@gmail.com> wrote: >>> >>> Thanks Koert, >>> >>> I did that before as well. Anyway this is dependencies >>> >>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" >>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" >>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" >>> >>> >>> and the error >>> >>> >>> [info] Compiling 1 Scala source to >>> /data6/hduser/scala/ETL_scratchpad_dummy/target/scala-2.10/classes... >>> [error] >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/scala/ETL_scratchpad_dummy.scala:4: >>> object hive is not a member of package org.apache.spark.sql >>> [error] import org.apache.spark.sql.hive.HiveContext >>> [error] ^ >>> [error] >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/scala/ETL_scratchpad_dummy.scala:20: >>> object hive is not a member of package org.apache.spark.sql >>> [error] val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >>> >>> >>> >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> Disclaimer: Use it at your own risk. Any and all responsibility for any >>> loss, damage or destruction of data or any other property which may arise >>> from relying on this email's technical content is explicitly disclaimed. The >>> author will in no case be liable for any monetary damages arising from such >>> loss, damage or destruction. >>> >>> >>> >>> >>> On 14 August 2016 at 17:00, Koert Kuipers <ko...@tresata.com> wrote: >>>> >>>> you cannot mix spark 1 and spark 2 jars >>>> >>>> change this >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>>> to >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" >>>> >>>> On Sun, Aug 14, 2016 at 11:58 AM, Mich Talebzadeh >>>> <mich.talebza...@gmail.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> In Spark 2 I am using sbt or mvn to compile my scala program. This used >>>>> to compile and run perfectly with Spark 1.6.1 but now it is throwing error >>>>> >>>>> >>>>> I believe the problem is here. I have >>>>> >>>>> name := "scala" >>>>> version := "1.0" >>>>> scalaVersion := "2.11.7" >>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" >>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" >>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>>>> >>>>> However the error I am getting is >>>>> >>>>> [error] bad symbolic reference. A signature in HiveContext.class refers >>>>> to type Logging >>>>> [error] in package org.apache.spark which is not available. >>>>> [error] It may be completely missing from the current classpath, or the >>>>> version on >>>>> [error] the classpath might be incompatible with the version used when >>>>> compiling HiveContext.class. >>>>> [error] one error found >>>>> [error] (compile:compileIncremental) Compilation failed >>>>> >>>>> >>>>> And this is the code >>>>> >>>>> import org.apache.spark.SparkContext >>>>> import org.apache.spark.SparkConf >>>>> import org.apache.spark.sql.Row >>>>> import org.apache.spark.sql.hive.HiveContext >>>>> import org.apache.spark.sql.types._ >>>>> import org.apache.spark.sql.SparkSession >>>>> import org.apache.spark.sql.functions._ >>>>> object ETL_scratchpad_dummy { >>>>> def main(args: Array[String]) { >>>>> val conf = new SparkConf(). >>>>> setAppName("ETL_scratchpad_dummy"). >>>>> set("spark.driver.allowMultipleContexts", "true"). >>>>> set("enableHiveSupport","true") >>>>> val sc = new SparkContext(conf) >>>>> //import sqlContext.implicits._ >>>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) >>>>> HiveContext.sql("use oraclehadoop") >>>>> >>>>> >>>>> Anyone has come across this >>>>> >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> Disclaimer: Use it at your own risk. Any and all responsibility for any >>>>> loss, damage or destruction of data or any other property which may arise >>>>> from relying on this email's technical content is explicitly disclaimed. >>>>> The >>>>> author will in no case be liable for any monetary damages arising from >>>>> such >>>>> loss, damage or destruction. >>>>> >>>>> >>>> >>>> >>> >> > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
