Re: Using Spark Streaming with Kafka 0.7.2

2014-07-29 Thread Andre Schumacher
Hi, For testing you could also just use the Kafka 0.7.2 console consumer and pipe it's output to netcat (nc) and process that as in the example https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala That worked for me. Back

Re: Using Spark Streaming with Kafka 0.7.2

2014-07-25 Thread Tathagata Das
Spark Streaming is built as part of the whole Spark repository. Hence follow Spark's building instructions to build Spark Streaming along with Spark. Spark Streaming 0.8.1 was built with kafka 0.7.2. You can take a look. If necessary, I

Using Spark Streaming with Kafka 0.7.2

2014-07-25 Thread maddenpj
Hi all, Currently we have Kafka 0.7.2 running in production and can't upgrade for external reasons however spark streaming (1.0.1) was built with Kafka 0.8.0. What is the best way to use spark streaming with older versions of Kafka. Currently I'm investigating trying to build spark streaming mysel