For you, it should be spark-core_2.10-1.5.1.jar Please replace version of Spark in my example with the version you use.
On Thu, Apr 21, 2016 at 1:23 PM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Ted > > I cannot see spark-core_2.11-2.0.0-SNAPSHOT.jar under > > https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/ > > Sorry where are these artefacts please? > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 21 April 2016 at 20:24, Ted Yu <yuzhih...@gmail.com> wrote: > >> Plug in 1.5.1 for your jars: >> >> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging >> ... >> 1781 Thu Apr 21 08:19:34 PDT 2016 >> org/apache/spark/internal/Logging$.class >> >> jar tvf >> external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep >> LeaderOffset >> ... >> 3310 Thu Apr 21 08:38:40 PDT 2016 >> org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class >> >> On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> These two are giving me griefs: >>> >>> scala> import org.apache.spark.internal.Logging >>> <console>:26: error: object internal is not a member of package >>> org.apache.spark >>> import org.apache.spark.internal.Logging >>> >>> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset >>> <console>:29: error: object KafkaCluster in package kafka cannot be >>> accessed in package org.apache.spark.streaming.kafka >>> import >>> org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset >>> >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >>> On 21 April 2016 at 18:29, Mich Talebzadeh <mich.talebza...@gmail.com> >>> wrote: >>> >>>> >>>> Thanks >>>> >>>> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite >>>> 1787 Wed Sep 23 23:34:26 BST 2015 >>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >>>> 1780 Wed Sep 23 23:34:26 BST 2015 >>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >>>> 3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class >>>> >>>> >>>> Dr Mich Talebzadeh >>>> >>>> >>>> >>>> LinkedIn * >>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> >>>> On 21 April 2016 at 18:21, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Please replace version number for the release you are using : >>>>> >>>>> spark-core_2.10-1.5.1-tests.jar >>>>> >>>>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> I don't seem to be able to locate >>>>>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :( >>>>>> >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> >>>>>> >>>>>> On 21 April 2016 at 17:45, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> It is in core-XX-tests jar: >>>>>>> >>>>>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | >>>>>>> grep SparkFunSuite >>>>>>> 1830 Thu Apr 21 08:19:14 PDT 2016 >>>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >>>>>>> 1823 Thu Apr 21 08:19:14 PDT 2016 >>>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >>>>>>> 6232 Thu Apr 21 08:19:14 PDT 2016 >>>>>>> org/apache/spark/SparkFunSuite.class >>>>>>> >>>>>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> like war of attrition :) >>>>>>>> >>>>>>>> now I get with sbt >>>>>>>> >>>>>>>> object SparkFunSuite is not a member of package org.apache.spark >>>>>>>> >>>>>>>> >>>>>>>> Dr Mich Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> LinkedIn * >>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 21 April 2016 at 17:22, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Have you tried the following ? >>>>>>>>> >>>>>>>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" >>>>>>>>> >>>>>>>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Unfortunately this sbt dependency is not working >>>>>>>>>> >>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % >>>>>>>>>> "1.5.1" % "provided" >>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % >>>>>>>>>> "1.5.1" % "provided" >>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % >>>>>>>>>> "1.5.1" % "provided" >>>>>>>>>> libraryDependencies += "junit" % "junit" % "4.12" >>>>>>>>>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0" >>>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % >>>>>>>>>> "1.6.1" % "provided" >>>>>>>>>> libraryDependencies += "org.apache.spark" %% >>>>>>>>>> "spark-streaming-kafka" % "1.6.1" >>>>>>>>>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6" >>>>>>>>>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" >>>>>>>>>> % "test"* >>>>>>>>>> >>>>>>>>>> Getting error >>>>>>>>>> >>>>>>>>>> [info] Compiling 1 Scala source to >>>>>>>>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes... >>>>>>>>>> [error] >>>>>>>>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32: >>>>>>>>>> object scalatest is not a member of package org >>>>>>>>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>> [error] ^ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> LinkedIn * >>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 21 April 2016 at 16:49, Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Ted. It was a typo in my alias and it is sorted now >>>>>>>>>>> >>>>>>>>>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 >>>>>>>>>>> --jars >>>>>>>>>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar' >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> LinkedIn * >>>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 21 April 2016 at 16:44, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I tried on refreshed copy of master branch: >>>>>>>>>>>> >>>>>>>>>>>> $ bin/spark-shell --jars >>>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>>>> ... >>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>>> >>>>>>>>>>>> BTW I noticed an extra leading comma after '--jars' in your >>>>>>>>>>>> email. >>>>>>>>>>>> Not sure if that matters. >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu <yuzhih...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Mich: >>>>>>>>>>>>> >>>>>>>>>>>>> $ jar tvf >>>>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>>>>> | grep BeforeAndAfter >>>>>>>>>>>>> 4257 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>>> org/scalatest/BeforeAndAfter$class.class >>>>>>>>>>>>> 2602 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>>> org/scalatest/BeforeAndAfter.class >>>>>>>>>>>>> 1998 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class >>>>>>>>>>>>> >>>>>>>>>>>>> FYI >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh < >>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> to Mario, Alonso, Luciano, user >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Following example in >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does anyone know which jar file this belongs to? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell >>>>>>>>>>>>>> >>>>>>>>>>>>>> spark-shell --master spark://50.140.197.217:7077 --jars >>>>>>>>>>>>>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar, >>>>>>>>>>>>>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'* >>>>>>>>>>>>>> >>>>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, >>>>>>>>>>>>>> BeforeAndAfterAll} >>>>>>>>>>>>>> <console>:28: error: object scalatest is not a member of >>>>>>>>>>>>>> package org >>>>>>>>>>>>>> import org.scalatest.{BeforeAndAfter, >>>>>>>>>>>>>> BeforeAndAfterAll} >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> LinkedIn * >>>>>>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
