Spark Streaming and Kafka MultiNode Setup - Data Locality

Hi All , Just wanted to find out if there is an benefits to installing kafka brokers and spark nodes on the same machine ? is it possible that spark can pull data from kafka if it is local to the node i.e. the broker or partition is on the same machine. Thanks, Ashish

Re: Spark Streaming and Kafka MultiNode Setup - Data Locality

seconds. -adrian From: Cody Koeninger <c...@koeninger.org> Sent: Monday, September 21, 2015 10:19 PM To: Ashish Soni Cc: user Subject: Re: Spark Streaming and Kafka MultiNode Setup - Data Locality The direct stream already uses the kafka leader for a

Re: Spark Streaming and Kafka MultiNode Setup - Data Locality

The direct stream already uses the kafka leader for a given partition as the preferred location. I don't run kafka on the same nodes as spark, and I don't know anyone who does, so that situation isn't particularly well tested. On Mon, Sep 21, 2015 at 1:15 PM, Ashish Soni