Any reason not to use SparkStreaming directly with HDFS files, so you'll get locality guarantees from the Hadoop framework? StreamContext has textFileStream() method you could use for this.
On Mon, Mar 16, 2015 at 12:46 PM, Daniel Haviv <daniel.ha...@veracity-group.com> wrote: > Hi, > Is it possible to assign specific partitions to specific nodes? > I want to upload files to HDFS, find out on which nodes the file resides > and then push their path into a topic and partition it by nodes. > This way I can ensure that the consumer (Spark Streaming) will consume both > the message and file locally. > > Can this be achieved ? > > Thanks, > Daniel
