Plug in 1.5.1 for your jars: $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging ... 1781 Thu Apr 21 08:19:34 PDT 2016 org/apache/spark/internal/Logging$.class
jar tvf external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep LeaderOffset ... 3310 Thu Apr 21 08:38:40 PDT 2016 org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh <mich.talebza...@gmail.com > wrote: > These two are giving me griefs: > > scala> import org.apache.spark.internal.Logging > <console>:26: error: object internal is not a member of package > org.apache.spark > import org.apache.spark.internal.Logging > > scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset > <console>:29: error: object KafkaCluster in package kafka cannot be > accessed in package org.apache.spark.streaming.kafka > import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 21 April 2016 at 18:29, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> >> Thanks >> >> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite >> 1787 Wed Sep 23 23:34:26 BST 2015 >> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >> 1780 Wed Sep 23 23:34:26 BST 2015 >> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >> 3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 21 April 2016 at 18:21, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Please replace version number for the release you are using : >>> >>> spark-core_2.10-1.5.1-tests.jar >>> >>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> I don't seem to be able to locate >>>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :( >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> On 21 April 2016 at 17:45, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> It is in core-XX-tests jar: >>>>> >>>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | >>>>> grep SparkFunSuite >>>>> 1830 Thu Apr 21 08:19:14 PDT 2016 >>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >>>>> 1823 Thu Apr 21 08:19:14 PDT 2016 >>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >>>>> 6232 Thu Apr 21 08:19:14 PDT 2016 >>>>> org/apache/spark/SparkFunSuite.class >>>>> >>>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> like war of attrition :) >>>>>> >>>>>> now I get with sbt >>>>>> >>>>>> object SparkFunSuite is not a member of package org.apache.spark >>>>>> >>>>>> >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> >>>>>> >>>>>> On 21 April 2016 at 17:22, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> Have you tried the following ? >>>>>>> >>>>>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" >>>>>>> >>>>>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> Unfortunately this sbt dependency is not working >>>>>>>> >>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" >>>>>>>> % "provided" >>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >>>>>>>> % "provided" >>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >>>>>>>> % "provided" >>>>>>>> libraryDependencies += "junit" % "junit" % "4.12" >>>>>>>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0" >>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % >>>>>>>> "1.6.1" % "provided" >>>>>>>> libraryDependencies += "org.apache.spark" %% >>>>>>>> "spark-streaming-kafka" % "1.6.1" >>>>>>>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6" >>>>>>>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % >>>>>>>> "test"* >>>>>>>> >>>>>>>> Getting error >>>>>>>> >>>>>>>> [info] Compiling 1 Scala source to >>>>>>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes... >>>>>>>> [error] >>>>>>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32: >>>>>>>> object scalatest is not a member of package org >>>>>>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>> [error] ^ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Dr Mich Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> LinkedIn * >>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 21 April 2016 at 16:49, Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks Ted. It was a typo in my alias and it is sorted now >>>>>>>>> >>>>>>>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 >>>>>>>>> --jars >>>>>>>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar' >>>>>>>>> >>>>>>>>> >>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Dr Mich Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> LinkedIn * >>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 21 April 2016 at 16:44, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I tried on refreshed copy of master branch: >>>>>>>>>> >>>>>>>>>> $ bin/spark-shell --jars >>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>> ... >>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>> >>>>>>>>>> BTW I noticed an extra leading comma after '--jars' in your email. >>>>>>>>>> Not sure if that matters. >>>>>>>>>> >>>>>>>>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu <yuzhih...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Mich: >>>>>>>>>>> >>>>>>>>>>> $ jar tvf >>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>>> | grep BeforeAndAfter >>>>>>>>>>> 4257 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>> org/scalatest/BeforeAndAfter$class.class >>>>>>>>>>> 2602 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>> org/scalatest/BeforeAndAfter.class >>>>>>>>>>> 1998 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class >>>>>>>>>>> >>>>>>>>>>> FYI >>>>>>>>>>> >>>>>>>>>>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> to Mario, Alonso, Luciano, user >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Following example in >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532 >>>>>>>>>>>> >>>>>>>>>>>> Does anyone know which jar file this belongs to? >>>>>>>>>>>> >>>>>>>>>>>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell >>>>>>>>>>>> >>>>>>>>>>>> spark-shell --master spark://50.140.197.217:7077 --jars >>>>>>>>>>>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar, >>>>>>>>>>>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'* >>>>>>>>>>>> >>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>>> <console>:28: error: object scalatest is not a member of >>>>>>>>>>>> package org >>>>>>>>>>>> import org.scalatest.{BeforeAndAfter, >>>>>>>>>>>> BeforeAndAfterAll} >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> LinkedIn * >>>>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
