Re: Hadoop streaming like feature for Spark

[Ewen Cheslack-Postava]

Take a look at RDD.pipe(). You could also accomplish the same thing using RDD.mapPartitions, which you pass a function that processes the iterator for each partition rather than processing each element individually. This lets you only start up as many processes as there are partitions, pipe the contents of each iterator to them, then collect the output. This might be useful if, e.g., your external process doesn't use line-oriented input/output. -Ewen Jaonary Rabarisoa March 20, 2014 at 1:04 AM Dear all, Dear all, Does Spark has a kind of Hadoop streaming feature to run external process to manipulate data from RDD sent through stdin and stdout ? Best, Jaonary