Iterating an RDD gives you each partition in order of their split index. I'd like to be able to get each partition in reverse order, but I'm having difficultly implementing the compute() method. I thought I could do something like this:
override def getDependencies: Seq[Dependency[_]] = { Seq(new NarrowDependency[T](prev) { def getParents(partitionId: Int): Seq[Int] = { Seq(prev.partitions.size - partitionId - 1) } }) } override def compute(split: Partition, context: TaskContext): Iterator[T] = { firstParent[T].iterator(split, context).toArray.reverseIterator } But that doesn't work. How do I get one split to depend on exactly one split from the parent that does not match indices?