Is the RDD partition index you get when you call mapPartitionWithIndex
consistent under fault-tolerance condition?

I.e.

1. Say index is 1 for one of the partitions when you call
data.mapPartitionWithIndex((index, rows) => ....) // Say index is 1
2. The partition fails (maybe a long with a bunch of other partitions).
3. When the partitions get restarted somewhere else, will they retain the
same index value, as well as all the lineage arguments?

Reply via email to