Iterating an RDD gives you each partition in order of their split index.
I'd like to be able to get each partition in reverse order, but I'm having
difficultly implementing the compute() method. I thought I could do
something like this:
override def getDependencies: Seq[Dependency[_]] = {
Seq(new NarrowDependency[T](prev) {
def getParents(partitionId: Int): Seq[Int] = {
Seq(prev.partitions.size - partitionId - 1)
}
})
}
override def compute(split: Partition, context: TaskContext): Iterator[T]
= {
firstParent[T].iterator(split, context).toArray.reverseIterator
}
But that doesn't work. How do I get one split to depend on exactly one
split from the parent that does not match indices?
