You could do a mapPartitions and Filter within each partition -- Something like
numbers <- parallelize(sc, 1:20, 4L) evenNumbers <- lapplyPartition(numbers, function(part) { Filter( function(x) { x%%2 == 0} , part) }) collect(evenNumbers) It should be simple to add this to the SparkR API as well -- Let me know if you want to send a PR ! Thanks Shivaram On Wed, Jan 29, 2014 at 5:45 PM, Justin Lent <justinl...@gmail.com> wrote: > any idea when the Scala filter() function equivalent will be availalbe > in the SparkR implementation? Or is there a simple way to implement it > in R with the existing functions? > > Thanks! > -Justin