[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864811#comment-15864811 ]
Saisai Shao commented on SPARK-19579: ------------------------------------- Spark Streaming Kafka Python API doesn't support Kafka 0.10 module. You could only use against Kafka 0.8.x. > spark-submit fails to run Kafka Stream python script > ---------------------------------------------------- > > Key: SPARK-19579 > URL: https://issues.apache.org/jira/browse/SPARK-19579 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 2.1.0 > Environment: Linux Ubuntu 16.10 64bit > Reporter: Piotr Nestorow > > Kafka Stream python script is executed but it fails with: > TypeError: 'JavaPackage' object is not callable > The Spark Kafka streaming jar is provided: > spark-streaming-kafka-0-10_2.11-2.1.0.jar > Kafka version: kafka_2.11-0.10.1.1 > In the conf/spark-defaults.conf: > spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0 > Also at runtime: > bin/spark-submit --jars > /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar > --driver-class-path > /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar > /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py > localhost:2181 example_topic > Also the Spark Kafka streaming example: > 'examples/src/main/python/streaming/kafka_wordcount.py' > can be used to check this problem. > Nothing helps... > Detailed output: > _________________ > Spark Streaming's Kafka libraries not found in class path. Try one of the > following. > 1. Include the Kafka library and its dependencies with in the > spark-submit command as > $ bin/spark-submit --packages > org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ... > 2. Download the JAR of the artifact from Maven Central > http://search.maven.org/, > Group Id = org.apache.spark, Artifact Id = > spark-streaming-kafka-0-8-assembly, Version = 2.1.0. > Then, include the jar in the spark-submit command as > $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ... > ________________________________________________________________________________________________ > Traceback (most recent call last): > File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line > 18, in <module> > kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", > {topic: 1}) > File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", > line 69, in createStream > File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", > line 195, in _get_helper > TypeError: 'JavaPackage' object is not callable -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org