Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/21758#discussion_r203733553
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1839,6 +1844,18 @@ abstract class RDD[T: ClassTag](
def toJavaRDD() : JavaRDD[T] = {
new JavaRDD(this)(elementClassTag)
}
+
+ /**
+ * Whether the RDD is in a barrier stage. Spark must launch all the
tasks at the same time for a
+ * barrier stage.
+ *
+ * An RDD is in a barrier stage, if at least one of its parent RDD(s),
or itself, are mapped from
+ * a RDDBarrier. This function always returns false for a
[[ShuffledRDD]], since a
+ * [[ShuffledRDD]] indicates start of a new stage.
+ */
+ def isBarrier(): Boolean = isBarrier_
+
+ @transient private lazy val isBarrier_ : Boolean =
dependencies.exists(_.rdd.isBarrier())
--- End diff --
This is a performance improvement to cache the value to avoid repeatedly
compute `isBarrier()` on a long RDD chain.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
