Hi Ted I cannot see spark-core_2.11-2.0.0-SNAPSHOT.jar under
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/ Sorry where are these artefacts please? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 21 April 2016 at 20:24, Ted Yu <yuzhih...@gmail.com> wrote: > Plug in 1.5.1 for your jars: > > $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT.jar | grep Logging > ... > 1781 Thu Apr 21 08:19:34 PDT 2016 > org/apache/spark/internal/Logging$.class > > jar tvf > external/kafka/target/spark-streaming-kafka_2.11-2.0.0-SNAPSHOT.jar | grep > LeaderOffset > ... > 3310 Thu Apr 21 08:38:40 PDT 2016 > org/apache/spark/streaming/kafka/KafkaCluster$LeaderOffset.class > > On Thu, Apr 21, 2016 at 11:52 AM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> These two are giving me griefs: >> >> scala> import org.apache.spark.internal.Logging >> <console>:26: error: object internal is not a member of package >> org.apache.spark >> import org.apache.spark.internal.Logging >> >> scala> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset >> <console>:29: error: object KafkaCluster in package kafka cannot be >> accessed in package org.apache.spark.streaming.kafka >> import org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset >> >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 21 April 2016 at 18:29, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> >>> Thanks >>> >>> jar tvf spark-core_2.10-1.5.1-tests.jar | grep SparkFunSuite >>> 1787 Wed Sep 23 23:34:26 BST 2015 >>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >>> 1780 Wed Sep 23 23:34:26 BST 2015 >>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >>> 3982 Wed Sep 23 23:34:26 BST 2015 org/apache/spark/SparkFunSuite.class >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >>> On 21 April 2016 at 18:21, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Please replace version number for the release you are using : >>>> >>>> spark-core_2.10-1.5.1-tests.jar >>>> >>>> On Thu, Apr 21, 2016 at 10:18 AM, Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> I don't seem to be able to locate >>>>> spark-core_2.11-2.0.0-SNAPSHOT-tests.jar file :( >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> >>>>> On 21 April 2016 at 17:45, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>>> It is in core-XX-tests jar: >>>>>> >>>>>> $ jar tvf ./core/target/spark-core_2.11-2.0.0-SNAPSHOT-tests.jar | >>>>>> grep SparkFunSuite >>>>>> 1830 Thu Apr 21 08:19:14 PDT 2016 >>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$1.class >>>>>> 1823 Thu Apr 21 08:19:14 PDT 2016 >>>>>> org/apache/spark/SparkFunSuite$$anonfun$withFixture$2.class >>>>>> 6232 Thu Apr 21 08:19:14 PDT 2016 >>>>>> org/apache/spark/SparkFunSuite.class >>>>>> >>>>>> On Thu, Apr 21, 2016 at 9:39 AM, Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> like war of attrition :) >>>>>>> >>>>>>> now I get with sbt >>>>>>> >>>>>>> object SparkFunSuite is not a member of package org.apache.spark >>>>>>> >>>>>>> >>>>>>> Dr Mich Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> LinkedIn * >>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://talebzadehmich.wordpress.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 21 April 2016 at 17:22, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>> >>>>>>>> Have you tried the following ? >>>>>>>> >>>>>>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" >>>>>>>> >>>>>>>> On Thu, Apr 21, 2016 at 9:19 AM, Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Unfortunately this sbt dependency is not working >>>>>>>>> >>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % >>>>>>>>> "1.5.1" % "provided" >>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % >>>>>>>>> "1.5.1" % "provided" >>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-hive" % >>>>>>>>> "1.5.1" % "provided" >>>>>>>>> libraryDependencies += "junit" % "junit" % "4.12" >>>>>>>>> libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0" >>>>>>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % >>>>>>>>> "1.6.1" % "provided" >>>>>>>>> libraryDependencies += "org.apache.spark" %% >>>>>>>>> "spark-streaming-kafka" % "1.6.1" >>>>>>>>> libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6" >>>>>>>>> *libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % >>>>>>>>> "test"* >>>>>>>>> >>>>>>>>> Getting error >>>>>>>>> >>>>>>>>> [info] Compiling 1 Scala source to >>>>>>>>> /data6/hduser/scala/CEP_assembly/target/scala-2.10/classes... >>>>>>>>> [error] >>>>>>>>> /data6/hduser/scala/CEP_assembly/src/main/scala/myPackage/CEP_assemly.scala:32: >>>>>>>>> object scalatest is not a member of package org >>>>>>>>> [error] import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>> [error] ^ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Dr Mich Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> LinkedIn * >>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 21 April 2016 at 16:49, Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks Ted. It was a typo in my alias and it is sorted now >>>>>>>>>> >>>>>>>>>> slong='rlwrap spark-shell --master spark://50.140.197.217:7077 >>>>>>>>>> --jars >>>>>>>>>> /home/hduser/jars/ojdbc6.jar,/home/hduser/jars/jconn4.jar,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar,/home/hduser/jars/scalatest_2.11-2.2.6.jar' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> LinkedIn * >>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 21 April 2016 at 16:44, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I tried on refreshed copy of master branch: >>>>>>>>>>> >>>>>>>>>>> $ bin/spark-shell --jars >>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>>> ... >>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>> >>>>>>>>>>> BTW I noticed an extra leading comma after '--jars' in your >>>>>>>>>>> email. >>>>>>>>>>> Not sure if that matters. >>>>>>>>>>> >>>>>>>>>>> On Thu, Apr 21, 2016 at 8:39 AM, Ted Yu <yuzhih...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Mich: >>>>>>>>>>>> >>>>>>>>>>>> $ jar tvf >>>>>>>>>>>> /home/hbase/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar >>>>>>>>>>>> | grep BeforeAndAfter >>>>>>>>>>>> 4257 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>> org/scalatest/BeforeAndAfter$class.class >>>>>>>>>>>> 2602 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>> org/scalatest/BeforeAndAfter.class >>>>>>>>>>>> 1998 Sat Dec 26 14:35:48 PST 2015 >>>>>>>>>>>> org/scalatest/BeforeAndAfterAll$$anonfun$run$1.class >>>>>>>>>>>> >>>>>>>>>>>> FYI >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Apr 21, 2016 at 8:35 AM, Mich Talebzadeh < >>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> to Mario, Alonso, Luciano, user >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Following example in >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://github.com/agsachin/spark/blob/CEP/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala#L532 >>>>>>>>>>>>> >>>>>>>>>>>>> Does anyone know which jar file this belongs to? >>>>>>>>>>>>> >>>>>>>>>>>>> I use *scalatest_2.11-2.2.6.jar *in my spark-shell >>>>>>>>>>>>> >>>>>>>>>>>>> spark-shell --master spark://50.140.197.217:7077 --jars >>>>>>>>>>>>> ,/home/hduser/jars/junit-4.12.jar,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar, >>>>>>>>>>>>> /*home/hduser/jars/scalatest_2.11-2.2.6.jar'* >>>>>>>>>>>>> >>>>>>>>>>>>> scala> import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} >>>>>>>>>>>>> <console>:28: error: object scalatest is not a member of >>>>>>>>>>>>> package org >>>>>>>>>>>>> import org.scalatest.{BeforeAndAfter, >>>>>>>>>>>>> BeforeAndAfterAll} >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> LinkedIn * >>>>>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
