We do - using Spark streaming, Kafka, HDFS all collocated on the same nodes. Works great so far.
Spark picks up the location information and reads data from the partitions hosted by the local broker, showing up as NODE_LOCAL in the UI. You also need to look at the locality options in the config (spark.locality.waitand friends) - just to make sure you're not wasting time if the kafka cluster becomes unbalanced and there are fewer cores than partitions on a particular node - you want to get to RACK_LOCAL as quickly as possible, we've set this to 500 milis instead of the default of 3 seconds. -adrian ________________________________ From: Cody Koeninger <c...@koeninger.org> Sent: Monday, September 21, 2015 10:19 PM To: Ashish Soni Cc: user Subject: Re: Spark Streaming and Kafka MultiNode Setup - Data Locality The direct stream already uses the kafka leader for a given partition as the preferred location. I don't run kafka on the same nodes as spark, and I don't know anyone who does, so that situation isn't particularly well tested. On Mon, Sep 21, 2015 at 1:15 PM, Ashish Soni <asoni.le...@gmail.com<mailto:asoni.le...@gmail.com>> wrote: Hi All , Just wanted to find out if there is an benefits to installing kafka brokers and spark nodes on the same machine ? is it possible that spark can pull data from kafka if it is local to the node i.e. the broker or partition is on the same machine. Thanks, Ashish
