LOL
well the issue here was the dependencies scripted in that shell
script which was modified to add "provided" to it.
The script itself still works just the content of one of functions had to
be edited
function create_sbt_file {
SBT_FILE=${GEN_APPSDIR}/scala/${APPLICATION}/${FILE_NAME}.sbt
[ -f ${SBT_FILE} ] && rm -f ${SBT_FILE}
cat >> $SBT_FILE << !
name := "scala"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" %
"provided"
.....
.....
!
}
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 14 August 2016 at 20:17, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi Mich,
>
> Yeah, you don't have to worry about it...and that's why you're asking
> these questions ;-)
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Sun, Aug 14, 2016 at 12:06 PM, Mich Talebzadeh
> <mich.talebza...@gmail.com> wrote:
> > The magic does all that(including compiling and submitting with the jar
> > file. It is flexible as it does all this for any Sala program. it creates
> > sub-directories, compiles, submits etc so I don't have to worry about it.
> >
> > HTH
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn
> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > Disclaimer: Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> The
> > author will in no case be liable for any monetary damages arising from
> such
> > loss, damage or destruction.
> >
> >
> >
> >
> > On 14 August 2016 at 20:01, Jacek Laskowski <ja...@japila.pl> wrote:
> >>
> >> Hi,
> >>
> >> You should have all the deps being "provided" since they're provided
> >> by spark infra after you spark-submit the uber-jar for the app.
> >>
> >> What's the "magic" in local.ksh? Why don't you sbt assembly and do
> >> spark-submit with the uber-jar?
> >>
> >> Pozdrawiam,
> >> Jacek Laskowski
> >> ----
> >> https://medium.com/@jaceklaskowski/
> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> >> Follow me at https://twitter.com/jaceklaskowski
> >>
> >>
> >> On Sun, Aug 14, 2016 at 11:52 AM, Mich Talebzadeh
> >> <mich.talebza...@gmail.com> wrote:
> >> > Thanks Jacek,
> >> >
> >> > I thought there was some dependency issue. This did the trick
> >> >
> >> > libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
> >> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
> >> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0" %
> >> > "provided"
> >> >
> >> > I use a shell script that builds the jar file depending on type (sbt,
> >> > mvn,
> >> > assembly) and submits it via spark-submit ..
> >> >
> >> > ./local.ksh -A ETL_scratchpad_dummy -T sbt
> >> >
> >> > As I understand "provided" means that the dependencies will be
> provided
> >> > at
> >> > run-time (spark-submit) through the jar files but they are not needed
> at
> >> > compile time.
> >> >
> >> > Having said that am I correct that error message below
> >> >
> >> > [error] bad symbolic reference. A signature in HiveContext.class
> refers
> >> > to
> >> > type Logging
> >> > [error] in package org.apache.spark which is not available.
> >> > [error] It may be completely missing from the current classpath, or
> the
> >> > version on
> >> > [error] the classpath might be incompatible with the version used when
> >> > compiling HiveContext.class.
> >> > [error] one error found
> >> > [error] (compile:compileIncremental) Compilation failed
> >> >
> >> > meant that some form of libraries incompatibility was happening at
> >> > compile
> >> > time?
> >> >
> >> > Cheers
> >> >
> >> >
> >> > Dr Mich Talebzadeh
> >> >
> >> >
> >> >
> >> > LinkedIn
> >> >
> >> > https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> >
> >> >
> >> >
> >> > http://talebzadehmich.wordpress.com
> >> >
> >> >
> >> > Disclaimer: Use it at your own risk. Any and all responsibility for
> any
> >> > loss, damage or destruction of data or any other property which may
> >> > arise
> >> > from relying on this email's technical content is explicitly
> disclaimed.
> >> > The
> >> > author will in no case be liable for any monetary damages arising from
> >> > such
> >> > loss, damage or destruction.
> >> >
> >> >
> >> >
> >> >
> >> > On 14 August 2016 at 19:11, Jacek Laskowski <ja...@japila.pl> wrote:
> >> >>
> >> >> Go to spark-shell and do :imports. You'll see all the imports and you
> >> >> could copy and paste them in your app. (but there are not many
> >> >> honestly and that won't help you much)
> >> >>
> >> >> HiveContext lives in spark-hive. You don't need spark-sql and
> >> >> spark-hive since the latter uses the former as a dependency (unless
> >> >> you're using types that come from the other dependencies). You don't
> >> >> need spark-core either. Make the dependencies simpler by:
> >> >>
> >> >> libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.0.0"
> >> >>
> >> >> and mark it % Provided.
> >> >>
> >> >> The reason for provided is that you don't need that for uber-jar that
> >> >> you're going to spark-submit.
> >> >>
> >> >> Don't forget to reload your session of sbt you're compiling in.
> Unsure
> >> >> how you do it so quit your sbt session and do `sbt compile`.
> >> >>
> >> >> Ask away if you need more details.
> >> >>
> >> >> Pozdrawiam,
> >> >> Jacek Laskowski
> >> >> ----
> >> >> https://medium.com/@jaceklaskowski/
> >> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> >> >> Follow me at https://twitter.com/jaceklaskowski
> >> >>
> >> >>
> >> >> On Sun, Aug 14, 2016 at 9:26 AM, Mich Talebzadeh
> >> >> <mich.talebza...@gmail.com> wrote:
> >> >> > The issue is on Spark shell this works OK
> >> >> >
> >> >> > Spark context Web UI available at http://50.140.197.217:55555
> >> >> > Spark context available as 'sc' (master = local, app id =
> >> >> > local-1471191662017).
> >> >> > Spark session available as 'spark'.
> >> >> > Welcome to
> >> >> > ____ __
> >> >> > / __/__ ___ _____/ /__
> >> >> > _\ \/ _ \/ _ `/ __/ '_/
> >> >> > /___/ .__/\_,_/_/ /_/\_\ version 2.0.0
> >> >> > /_/
> >> >> > Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java
> >> >> > 1.8.0_77)
> >> >> > Type in expressions to have them evaluated.
> >> >> > Type :help for more information.
> >> >> > scala> import org.apache.spark.SparkContext
> >> >> > scala> import org.apache.spark.SparkConf
> >> >> > scala> import org.apache.spark.sql.Row
> >> >> > scala> import org.apache.spark.sql.hive.HiveContext
> >> >> > scala> import org.apache.spark.sql.types._
> >> >> > scala> import org.apache.spark.sql.SparkSession
> >> >> > scala> import org.apache.spark.sql.functions._
> >> >> >
> >> >> > The code itself
> >> >> >
> >> >> >
> >> >> > scala> val conf = new SparkConf().
> >> >> > | setAppName("ETL_scratchpad_dummy").
> >> >> > | set("spark.driver.allowMultipleContexts",
> >> >> > "true").
> >> >> > | set("enableHiveSupport","true")
> >> >> > conf: org.apache.spark.SparkConf =
> >> >> > org.apache.spark.SparkConf@33215ffb
> >> >> >
> >> >> > scala> val sc = new SparkContext(conf)
> >> >> > sc: org.apache.spark.SparkContext =
> >> >> > org.apache.spark.SparkContext@3cbfdf5c
> >> >> >
> >> >> > scala> val HiveContext = new
> >> >> > org.apache.spark.sql.hive.HiveContext(sc)
> >> >> > warning: there was one deprecation warning; re-run with
> -deprecation
> >> >> > for
> >> >> > details
> >> >> > HiveContext: org.apache.spark.sql.hive.HiveContext =
> >> >> > org.apache.spark.sql.hive.HiveContext@2152fde5
> >> >> >
> >> >> > scala> HiveContext.sql("use oraclehadoop")
> >> >> > res0: org.apache.spark.sql.DataFrame = []
> >> >> >
> >> >> > I think I am getting something missing here a dependency
> >> >> >
> >> >> >
> >> >> > Dr Mich Talebzadeh
> >> >> >
> >> >> >
> >> >> >
> >> >> > LinkedIn
> >> >> >
> >> >> >
> >> >> > https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> >> >
> >> >> >
> >> >> >
> >> >> > http://talebzadehmich.wordpress.com
> >> >> >
> >> >> >
> >> >> > Disclaimer: Use it at your own risk. Any and all responsibility for
> >> >> > any
> >> >> > loss, damage or destruction of data or any other property which may
> >> >> > arise
> >> >> > from relying on this email's technical content is explicitly
> >> >> > disclaimed.
> >> >> > The
> >> >> > author will in no case be liable for any monetary damages arising
> >> >> > from
> >> >> > such
> >> >> > loss, damage or destruction.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 14 August 2016 at 17:16, Koert Kuipers <ko...@tresata.com>
> wrote:
> >> >> >>
> >> >> >> HiveContext is gone
> >> >> >>
> >> >> >> SparkSession now combines functionality of SqlContext and
> >> >> >> HiveContext
> >> >> >> (if
> >> >> >> hive support is available)
> >> >> >>
> >> >> >> On Sun, Aug 14, 2016 at 12:12 PM, Mich Talebzadeh
> >> >> >> <mich.talebza...@gmail.com> wrote:
> >> >> >>>
> >> >> >>> Thanks Koert,
> >> >> >>>
> >> >> >>> I did that before as well. Anyway this is dependencies
> >> >> >>>
> >> >> >>> libraryDependencies += "org.apache.spark" %% "spark-core" %
> "2.0.0"
> >> >> >>> libraryDependencies += "org.apache.spark" %% "spark-sql" %
> "2.0.0"
> >> >> >>> libraryDependencies += "org.apache.spark" %% "spark-hive" %
> "2.0.0"
> >> >> >>>
> >> >> >>>
> >> >> >>> and the error
> >> >> >>>
> >> >> >>>
> >> >> >>> [info] Compiling 1 Scala source to
> >> >> >>>
> >> >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/target/scala-
> 2.10/classes...
> >> >> >>> [error]
> >> >> >>>
> >> >> >>>
> >> >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/
> scala/ETL_scratchpad_dummy.scala:4:
> >> >> >>> object hive is not a member of package org.apache.spark.sql
> >> >> >>> [error] import org.apache.spark.sql.hive.HiveContext
> >> >> >>> [error] ^
> >> >> >>> [error]
> >> >> >>>
> >> >> >>>
> >> >> >>> /data6/hduser/scala/ETL_scratchpad_dummy/src/main/
> scala/ETL_scratchpad_dummy.scala:20:
> >> >> >>> object hive is not a member of package org.apache.spark.sql
> >> >> >>> [error] val HiveContext = new
> >> >> >>> org.apache.spark.sql.hive.HiveContext(sc)
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> Dr Mich Talebzadeh
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> LinkedIn
> >> >> >>>
> >> >> >>>
> >> >> >>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> http://talebzadehmich.wordpress.com
> >> >> >>>
> >> >> >>>
> >> >> >>> Disclaimer: Use it at your own risk. Any and all responsibility
> for
> >> >> >>> any
> >> >> >>> loss, damage or destruction of data or any other property which
> may
> >> >> >>> arise
> >> >> >>> from relying on this email's technical content is explicitly
> >> >> >>> disclaimed. The
> >> >> >>> author will in no case be liable for any monetary damages arising
> >> >> >>> from
> >> >> >>> such
> >> >> >>> loss, damage or destruction.
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On 14 August 2016 at 17:00, Koert Kuipers <ko...@tresata.com>
> >> >> >>> wrote:
> >> >> >>>>
> >> >> >>>> you cannot mix spark 1 and spark 2 jars
> >> >> >>>>
> >> >> >>>> change this
> >> >> >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" %
> >> >> >>>> "1.5.1"
> >> >> >>>> to
> >> >> >>>> libraryDependencies += "org.apache.spark" %% "spark-hive" %
> >> >> >>>> "2.0.0"
> >> >> >>>>
> >> >> >>>> On Sun, Aug 14, 2016 at 11:58 AM, Mich Talebzadeh
> >> >> >>>> <mich.talebza...@gmail.com> wrote:
> >> >> >>>>>
> >> >> >>>>> Hi,
> >> >> >>>>>
> >> >> >>>>> In Spark 2 I am using sbt or mvn to compile my scala program.
> >> >> >>>>> This
> >> >> >>>>> used
> >> >> >>>>> to compile and run perfectly with Spark 1.6.1 but now it is
> >> >> >>>>> throwing
> >> >> >>>>> error
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> I believe the problem is here. I have
> >> >> >>>>>
> >> >> >>>>> name := "scala"
> >> >> >>>>> version := "1.0"
> >> >> >>>>> scalaVersion := "2.11.7"
> >> >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-core" %
> >> >> >>>>> "2.0.0"
> >> >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" %
> >> >> >>>>> "2.0.0"
> >> >> >>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" %
> >> >> >>>>> "1.5.1"
> >> >> >>>>>
> >> >> >>>>> However the error I am getting is
> >> >> >>>>>
> >> >> >>>>> [error] bad symbolic reference. A signature in
> HiveContext.class
> >> >> >>>>> refers
> >> >> >>>>> to type Logging
> >> >> >>>>> [error] in package org.apache.spark which is not available.
> >> >> >>>>> [error] It may be completely missing from the current
> classpath,
> >> >> >>>>> or
> >> >> >>>>> the
> >> >> >>>>> version on
> >> >> >>>>> [error] the classpath might be incompatible with the version
> used
> >> >> >>>>> when
> >> >> >>>>> compiling HiveContext.class.
> >> >> >>>>> [error] one error found
> >> >> >>>>> [error] (compile:compileIncremental) Compilation failed
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> And this is the code
> >> >> >>>>>
> >> >> >>>>> import org.apache.spark.SparkContext
> >> >> >>>>> import org.apache.spark.SparkConf
> >> >> >>>>> import org.apache.spark.sql.Row
> >> >> >>>>> import org.apache.spark.sql.hive.HiveContext
> >> >> >>>>> import org.apache.spark.sql.types._
> >> >> >>>>> import org.apache.spark.sql.SparkSession
> >> >> >>>>> import org.apache.spark.sql.functions._
> >> >> >>>>> object ETL_scratchpad_dummy {
> >> >> >>>>> def main(args: Array[String]) {
> >> >> >>>>> val conf = new SparkConf().
> >> >> >>>>> setAppName("ETL_scratchpad_dummy").
> >> >> >>>>> set("spark.driver.allowMultipleContexts",
> "true").
> >> >> >>>>> set("enableHiveSupport","true")
> >> >> >>>>> val sc = new SparkContext(conf)
> >> >> >>>>> //import sqlContext.implicits._
> >> >> >>>>> val HiveContext = new org.apache.spark.sql.hive.
> HiveContext(sc)
> >> >> >>>>> HiveContext.sql("use oraclehadoop")
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Anyone has come across this
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Dr Mich Talebzadeh
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> LinkedIn
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> http://talebzadehmich.wordpress.com
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> Disclaimer: Use it at your own risk. Any and all responsibility
> >> >> >>>>> for
> >> >> >>>>> any
> >> >> >>>>> loss, damage or destruction of data or any other property which
> >> >> >>>>> may
> >> >> >>>>> arise
> >> >> >>>>> from relying on this email's technical content is explicitly
> >> >> >>>>> disclaimed. The
> >> >> >>>>> author will in no case be liable for any monetary damages
> arising
> >> >> >>>>> from such
> >> >> >>>>> loss, damage or destruction.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>
> >> >> >>
> >> >> >
> >> >
> >> >
> >
> >
>
