jerrypeng commented on code in PR #59523:
URL: https://github.com/apache/spark/pull/59523#discussion_r4291101625
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1489,6 +1489,28 @@ private[spark] class DAGScheduler(
checkPipelinedStageCreation(finalRDD, partitions, jobId)
}
+ /** Reject persistence that could bypass a native pipelined shuffle reader.
*/
+ private def checkPipelinedCacheSupported(finalRDD: RDD[_]): Unit = {
+ val persisted = new HashSet[RDD[_]]
+ traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+ if (rdd.getStorageLevel != StorageLevel.NONE ||
+
rdd.checkpointData.exists(_.isInstanceOf[LocalRDDCheckpointData[_]])) {
+ persisted += rdd
+ }
+ rdd.dependencies.foreach(dep => enqueue(dep.rdd))
+ }
+ // Cache locations cannot establish a stable cut: eviction can revive an
upstream producer,
+ // and a partial cache hit skips a reader whose producer still needs its
termination ACK.
+ // Check storage intent instead, including pending local checkpoints and
narrow descendants.
+ val visited = new HashSet[RDD[_]]
+ if (persisted.exists(rdd => rddChainReadsPipelinedShuffle(rdd, visited))) {
+ throw pipelinedUnsupportedError(
+ "cache/persist/localCheckpoint in a pipelined consumer's within-stage
chain; " +
+ "unpersist cached RDDs or rebuild locally checkpointed consumers
without " +
+ "localCheckpoint before submitting the job; persist producer inputs
instead")
Review Comment:
Agreed. In `A -PSD1-> B -PSD2-> C`, B also consumes PSD1, so “persist
producer inputs instead” was too broad.
Fixed in 009e4bfb318. The error now says “persist only RDDs that do not read
a pipelined shuffle within their stage”, while retaining the unpersist/rebuild
guidance in parentheses. The existing intermediate-stage rejection test now
asserts this qualified advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
