jerrypeng commented on code in PR #59523:
URL: https://github.com/apache/spark/pull/59523#discussion_r4408196498
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1516,6 +1515,29 @@ private[spark] class DAGScheduler(
checkPipelinedStageCreation(finalRDD, partitions, jobId)
}
+ /** Reject persistence that could bypass a native pipelined shuffle reader.
*/
+ private def checkPipelinedCacheSupported(finalRDD: RDD[_]): Unit = {
+ val persisted = new HashSet[RDD[_]]
+ traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+ if (rdd.getStorageLevel != StorageLevel.NONE ||
+
rdd.checkpointData.exists(_.isInstanceOf[LocalRDDCheckpointData[_]])) {
+ persisted += rdd
+ }
+ rdd.dependencies.foreach(dep => enqueue(dep.rdd))
+ }
+ // Cache locations cannot establish a stable cut: eviction can revive an
upstream producer,
+ // and a partial cache hit skips a reader whose producer still needs its
termination ACK.
+ // Check storage intent instead, including pending local checkpoints and
narrow descendants.
+ val visited = new HashSet[RDD[_]]
+ if (persisted.exists(rdd => rddChainReadsPipelinedShuffle(rdd, visited))) {
+ throw pipelinedUnsupportedError(
+ "cache/persist/localCheckpoint in a pipelined consumer's within-stage
chain " +
Review Comment:
Agreed. The proposed change uses find to retain the offending RDD, includes
its toString in the error, and directs the recovery advice at that RDD. The
regression test includes allowed cached producer inputs and verifies that the
error identifies the rejected consumer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
