bq. its Spark libs are all at 2.10 Clarification: 2.10 is version of Scala Your Spark version is 1.1.0
You can use earlier release of Kafka. Cheers On Mon, May 4, 2015 at 2:39 PM, Eric Ho <eric...@intel.com> wrote: > I still prefer to use Spark core / streaming /... at 2.10 becuase my DSE > is at 4.6.1 and its Spark libs are all at 2.10 ... My Scala code will run > on DSE machines which have Spark enabled. > > So, should I grab my Kakfa server at here ? > https://archive.apache.org/dist/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz > > On Mon, May 4, 2015 at 1:07 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Looks like you're using Spark 1.1.0 >> >> Support for Kafka 0.8.2 was added by: >> https://issues.apache.org/jira/browse/SPARK-2808 >> >> which would come in Spark 1.4.0 >> >> FYI >> >> On Mon, May 4, 2015 at 12:22 PM, Eric Ho <eric...@intel.com> wrote: >> >>> I'm submitting this via 'dse spark-submit' but somehow, I don't see any >>> loggings in my cluster or worker machines... >>> >>> How can I find out ? >>> >>> My cluster is running DSE 4.6.1 with Spark enabled. >>> My source is running Kafka 0.8.2.0 >>> >>> I'm launching my program on one of my DSE machines. >>> >>> Any insights much appreciated. >>> >>> Thanks. >>> >>> --------- >>> cas1.dev% dse spark-submit --verbose --deploy-mode cluster --master >>> spark://cas1.dev.kno.com:7077 --class >>> com.kno.highlights.counting.service.HighlightConsumer --driver-class-path >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> --driver-library-path >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> --properties-file /tmp/highlights-counting.properties >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar >>> --name HighlightConsumer >>> Using properties file: /tmp/highlights-counting.properties >>> Warning: Ignoring non-spark config property: >>> checkpoint_directory=checkpointForHighlights >>> Warning: Ignoring non-spark config property: zookeeper_port=2181 >>> Warning: Ignoring non-spark config property: >>> default_num_of_cores_per_topic=1 >>> Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 >>> Warning: Ignoring non-spark config property: >>> kafka_consumer_group=highlight_consumer_group >>> Warning: Ignoring non-spark config property: app_name=HighlightConsumer >>> Warning: Ignoring non-spark config property: >>> cassandra_keyspace=bookevents >>> Warning: Ignoring non-spark config property: scheduler_mode=FIFO >>> Warning: Ignoring non-spark config property: >>> highlight_topic=highlight_topic >>> Warning: Ignoring non-spark config property: cassandra_host= >>> cas1.dev.kno.com >>> Warning: Ignoring non-spark config property: checkpoint_interval=3 >>> Warning: Ignoring non-spark config property: zookeeper_host= >>> cas1.dev.kno.com >>> Adding default property: spark_master=spark://cas1.dev.kno.com:7077 >>> Warning: Ignoring non-spark config property: streaming_window=10 >>> Using properties file: /tmp/highlights-counting.properties >>> Warning: Ignoring non-spark config property: >>> checkpoint_directory=checkpointForHighlights >>> Warning: Ignoring non-spark config property: zookeeper_port=2181 >>> Warning: Ignoring non-spark config property: >>> default_num_of_cores_per_topic=1 >>> Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 >>> Warning: Ignoring non-spark config property: >>> kafka_consumer_group=highlight_consumer_group >>> Warning: Ignoring non-spark config property: app_name=HighlightConsumer >>> Warning: Ignoring non-spark config property: >>> cassandra_keyspace=bookevents >>> Warning: Ignoring non-spark config property: scheduler_mode=FIFO >>> Warning: Ignoring non-spark config property: >>> highlight_topic=highlight_topic >>> Warning: Ignoring non-spark config property: cassandra_host= >>> cas1.dev.kno.com >>> Warning: Ignoring non-spark config property: checkpoint_interval=3 >>> Warning: Ignoring non-spark config property: zookeeper_host= >>> cas1.dev.kno.com >>> Adding default property: spark_master=spark://cas1.dev.kno.com:7077 >>> Warning: Ignoring non-spark config property: streaming_window=10 >>> Parsed arguments: >>> master spark://cas1.dev.kno.com:7077 >>> deployMode cluster >>> executorMemory null >>> executorCores null >>> totalExecutorCores null >>> propertiesFile /tmp/highlights-counting.properties >>> extraSparkProperties Map() >>> driverMemory null >>> driverCores null >>> driverExtraClassPath >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> driverExtraLibraryPath >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> driverExtraJavaOptions null >>> supervise false >>> queue null >>> numExecutors null >>> files null >>> pyFiles null >>> archives null >>> mainClass >>> com.kno.highlights.counting.service.HighlightConsumer >>> primaryResource >>> >>> file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar >>> name >>> com.kno.highlights.counting.service.HighlightConsumer >>> childArgs [--name HighlightConsumer] >>> jars null >>> verbose true >>> >>> Default properties from /tmp/highlights-counting.properties: >>> spark_master -> spark://cas1.dev.kno.com:7077 >>> >>> >>> Using properties file: /tmp/highlights-counting.properties >>> Warning: Ignoring non-spark config property: >>> checkpoint_directory=checkpointForHighlights >>> Warning: Ignoring non-spark config property: zookeeper_port=2181 >>> Warning: Ignoring non-spark config property: >>> default_num_of_cores_per_topic=1 >>> Warning: Ignoring non-spark config property: num_of_concurrent_streams=2 >>> Warning: Ignoring non-spark config property: >>> kafka_consumer_group=highlight_consumer_group >>> Warning: Ignoring non-spark config property: app_name=HighlightConsumer >>> Warning: Ignoring non-spark config property: >>> cassandra_keyspace=bookevents >>> Warning: Ignoring non-spark config property: scheduler_mode=FIFO >>> Warning: Ignoring non-spark config property: >>> highlight_topic=highlight_topic >>> Warning: Ignoring non-spark config property: cassandra_host= >>> cas1.dev.kno.com >>> Warning: Ignoring non-spark config property: checkpoint_interval=3 >>> Warning: Ignoring non-spark config property: zookeeper_host= >>> cas1.dev.kno.com >>> Adding default property: spark_master=spark://cas1.dev.kno.com:7077 >>> Warning: Ignoring non-spark config property: streaming_window=10 >>> Main class: >>> org.apache.spark.deploy.Client >>> Arguments: >>> launch >>> spark://cas1.dev.kno.com:7077 >>> >>> file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar >>> com.kno.highlights.counting.service.HighlightConsumer >>> --name >>> HighlightConsumer >>> System properties: >>> spark_master -> spark://cas1.dev.kno.com:7077 >>> spark.driver.extraLibraryPath -> >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> SPARK_SUBMIT -> true >>> spark.app.name -> com.kno.highlights.counting.service.HighlightConsumer >>> spark.jars -> >>> >>> file:/opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib/kno-highlights-counting-service.kno-highlights-counting-service-0.1.jar >>> spark.master -> spark://cas1.dev.kno.com:7077 >>> spark.driver.extraClassPath -> >>> >>> /opt/kno/kno-highlights-counting-service/kno-highlights-counting-service-0.1/lib >>> Classpath elements: >>> >>> >>> >>> Sending launch command to spark://cas1.dev.kno.com:7077 >>> >>> ------------ >>> >>> Here's my build.sbt: >>> --------- >>> import AssemblyKeys._ >>> import NativePackagerHelper._ >>> >>> assemblySettings >>> >>> jarName in assembly := "kno-highlights-counting-service.jar" >>> >>> name := "kno-highlights-counting-service" >>> >>> version := "0.1" >>> >>> scalaVersion := "2.10.4" >>> >>> exportJars := true >>> >>> enablePlugins(JavaServerAppPackaging) >>> >>> resolvers ++= Seq( >>> "spray repo" at "http://repo.spray.io/";, >>> "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"; >>> ) >>> >>> packageOptions in (Compile, packageBin) += >>> Package.ManifestAttributes( java.util.jar.Attributes.Name.SEALED -> >>> "true" ) >>> >>> libraryDependencies ++= { >>> Seq( >>> // "org.apache.kafka" % "kafka_2.10" % "0.8.0" >>> exclude("org.jboss.netty", >>> "netty"), >>> "org.apache.spark" % "spark-core_2.10" % "1.1.0" >>> exclude("org.jboss.netty", "netty"), >>> "org.apache.spark" % "spark-streaming_2.10" % "1.1.0" >>> exclude("org.jboss.netty", "netty"), >>> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.1.0" >>> exclude("org.jboss.netty", "netty"), >>> "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0" >>> exclude("org.jboss.netty", "netty"), >>> "commons-io" % "commons-io" % "2.4", >>> "org.apache.commons" % "commons-pool2" % "2.3", >>> "ch.qos.logback" % "logback-classic" % "1.1.2", >>> "io.spray" %% "spray-json" % "1.3.1", >>> "com.typesafe" % "config" % "1.2.1" >>> ) >>> } >>> >>> seq(Revolver.settings: _*) >>> >>> atmosSettings >>> ------- >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/No-logs-from-my-cluster-worker-running-DSE-4-6-1-tp22759.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> > > > -- > > Intel Education > >
