Re: RESOLVED - Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

OK so this was Kafka issue? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 7 June 2016 at 16:55,

RESOLVED - Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Dear all, I managed to resolve the issue. Since I kept getting the exception "org.apache.spark.SparkException: java.nio.channels.ClosedChannelException”, a reasonable direction was checking the advertised.host.name key which as I’ve read from the docs basically sets for the broker the

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Dear Todd, By running bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic --broker-list localhost:9092 --time -1 I get the following current offset for :0:1760 But I guess this does not provide as much information. To answer your other question, onto how exactly do I track the offset

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Hi Dominik, Right, and spark 1.6.x uses Kafka v0.8.2.x as I recall. However, it appears as though the v.0.8 consumer is compatible with the Kafka v0.9.x broker, but not the other way around; sorry for the confusion there. With the direct stream, simple consumer, offsets are tracked by Spark

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.0" libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1" Please take a look at the SBT copy. I would rather think that

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

What version of Spark are you using? I do not believe that 1.6.x is compatible with 0.9.0.1 due to changes in the kafka clients between 0.8.2.2 and 0.9.0.x. See this for more information: https://issues.apache.org/jira/browse/SPARK-12177 -Todd On Tue, Jun 7, 2016 at 7:35 AM, Dominik Safaric

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Hi, Correct, I am using the 0.9.0.1 version. As already described, the topic contains messages. Those messages are produced using the Confluence REST API. However, what I’ve observed is that the problem is not in the Spark configuration, but rather Zookeeper or Kafka related. Take a look

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Hi, What's the version of Spark? You're using Kafka 0.9.0.1, ain't you? What's the topic name? Jacek On 7 Jun 2016 11:06 a.m., "Dominik Safaric" wrote: > As I am trying to integrate Kafka into Spark, the following exception > occurs: > >

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

For now you can move away from Spark and look at the cause of your kafka publishing Also check that zookeeper is running jps *17102* QuorumPeerMain runs on default port 2181 netstat -plten|grep 2181 tcp0 0 :::2181 :::* LISTEN 1005 8765628

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

> Sounds like the issue is with Kafka channel, it is closing. Made the same conclusion as well. I’ve even tried further refining the configuration files: Zookeeper properties: # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Sounds like the issue is with Kafka channel, it is closing. Reconnect due to socket error: java.nio.channels.ClosedChannelException Can you relax that val ssc = new StreamingContext(sparkConf, Seconds(20) Also how are you getting your source data? You can actually have both Spark and the

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Unfortunately, even with this Spark configuration and Kafka parameters, the same exception keeps occurring: 16/06/07 12:26:11 INFO SimpleConsumer: Reconnect due to socket error: java.nio.channels.ClosedChannelException org.apache.spark.SparkException: java.nio.channels.ClosedChannelException

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

ok that is good Yours is basically simple streaming with Kafka (publishing topic) and your Spark streaming. use the following as blueprint // Create a local StreamingContext with two working thread and batch interval of 2 seconds. val sparkConf = new SparkConf().

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

Dear Mich, Thank you for the reply. By running the following command in the command line: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic --from-beginning I do indeed retrieve all messages of a topic. Any indication onto what might cause the issue? An important note to

Re: Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

I assume you zookeeper is up and running can you confirm that you are getting topics from kafka independently for example on the command line ${KAFKA_HOME}/bin/kafka-console-consumer.sh --zookeeper rhes564:2181 --from-beginning --topic newtopic Dr Mich Talebzadeh LinkedIn *

Apache Spark Kafka Integration - org.apache.spark.SparkException: Couldn't find leader offsets for Set()

As I am trying to integrate Kafka into Spark, the following exception occurs: org.apache.spark.SparkException: java.nio.channels.ClosedChannelException org.apache.spark.SparkException: Couldn't find leader offsets for Set([**,0]) at