Github user squito commented on the issue: https://github.com/apache/spark/pull/21927 > Second thought: PartitionPruningRDD is just an implementation of RDD. Every user / developer can implement a similar one. Also this doesn't handle the case mentioned by @felixcheung : a.union(b).barrier(). So I'm thinking about checking number of partitions instead of instances of PartitionPruningRDD in this PR. Basically, we check the input RDD and all its parents have the same number of partitions. If not, we throw an error message like "Barrier execution mode doesn't support partition union / pruning.". Thoughts? yeah thats a good point, but what about `coalesce()`?? that should actually work, shouldn't it? Maybe you'd add an exception for `CoalescedRDD`, or add another property for `processAllInputPartitions` or something ...
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org