I am trying to read stored tweets in a file and send it to Kafka using Spark python.
The code is very simple but it does not work. The spark job runs correctly but nothing sent to Kafka Here is the code /#!/usr/bin/python # -*- coding: utf-8 -*- from pyspark import SparkContext, SparkConf from kafka import KafkaProducer import sys reload(sys) sys.setdefaultencoding('utf8') topic = "testTopic" if __name__ == "__main__": conf = SparkConf().setAppName("TestSparkFromPython") sc = SparkContext(conf=conf) producer = KafkaProducer(bootstrap_servers="10.62.54.111:9092") tweets = sc.textFile("/home/fanooos/Desktop/historical_scripts/output/1/activities_201603270430_201603270440.json") tweetsCollection = tweets.collect() for tweet in tweetsCollection: producer.send('testTopic', value=bytes(tweet))/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Sending-events-to-Kafka-from-spark-job-tp26622.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org