Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21758#discussion_r205250352
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1839,6 +1847,20 @@ abstract class RDD[T: ClassTag](
def toJavaRDD() : JavaRDD[T] = {
new JavaRDD(this)(elementClassTag)
}
+
+ /**
+ * Whether the RDD is in a barrier stage. Spark must launch all the
tasks at the same time for a
+ * barrier stage.
+ *
+ * An RDD is in a barrier stage, if at least one of its parent RDD(s),
or itself, are mapped from
+ * a RDDBarrier. This function always returns false for a
[[ShuffledRDD]], since a
+ * [[ShuffledRDD]] indicates start of a new stage.
+ */
+ private[spark] def isBarrier(): Boolean = isBarrier_
+
+ // From performance concern, cache the value to avoid repeatedly compute
`isBarrier()` on a long
+ // RDD chain.
+ @transient protected lazy val isBarrier_ : Boolean =
dependencies.exists(_.rdd.isBarrier())
--- End diff --
you need to explain why mappartitionsrdd has a different isBarrier
implementation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
