jerrypeng commented on code in PR #59523:
URL: https://github.com/apache/spark/pull/59523#discussion_r4429427544
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1520,13 +1519,39 @@ private[spark] class DAGScheduler(
checkPipelinedStageCreation(finalRDD, partitions, jobId)
}
+ /** Reject persistence that could bypass a native pipelined shuffle reader.
*/
+ private def checkPipelinedCacheSupported(finalRDD: RDD[_]): Unit = {
+ val persisted = new HashSet[RDD[_]]
+ traverseRDDGraph(finalRDD) { (rdd, enqueue) =>
+ if (rdd.getStorageLevel != StorageLevel.NONE ||
rdd.isLocallyCheckpointed) {
+ persisted += rdd
+ }
+ rdd.dependencies.foreach {
+ case pd: PipelinedShuffleDependency[_, _, _] => enqueue(pd.rdd)
+ case _: ShuffleDependency[_, _, _] =>
+ // Shape validation excludes pipelined readers below a regular
boundary.
+ case dep => enqueue(dep.rdd)
+ }
+ }
+ // Cache locations cannot establish a stable cut: eviction can revive an
upstream producer,
+ // and a partial cache hit skips a reader whose producer still needs its
termination ACK.
+ // Check storage intent instead, including pending local checkpoints and
narrow descendants.
+ val visited = new HashSet[RDD[_]]
+ persisted.find(rdd => rddChainReadsPipelinedShuffle(rdd, visited)).foreach
{ rdd =>
Review Comment:
Changed `persisted` to `ListBuffer`, since `traverseRDDGraph` already visits
each RDD once. This preserves traversal order while retaining the shared
visited set and short-circuiting `find`.
The regression uses a cached consumer and cached descendant with swapped
fixed hashes: it names the descendant first, then the consumer when the
descendant is unpersisted and resubmitted. The regression fails with the old
collector in both scheduler variants and passes with the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
