cloud-fan commented on code in PR #57286:
URL: https://github.com/apache/spark/pull/57286#discussion_r3616562554
##########
core/src/main/scala/org/apache/spark/Dependency.scala:
##########
@@ -258,6 +261,71 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
}
+/**
+ * :: DeveloperApi ::
+ * A [[ShuffleDependency]] whose output can be read incrementally: a consumer
stage may begin
+ * reading the shuffle output while the producer stage is still running,
rather than waiting for the
+ * producer's full, materialized output.
+ *
+ * This is a subtype of [[ShuffleDependency]] -- and thus, like it, a
first-class dependency kind
+ * alongside [[NarrowDependency]] under [[Dependency]]. It is intended to be
the marker the
+ * `DAGScheduler` will use to decide that the producer and consumer
+ * stages connected by this edge may run concurrently (a "pipelined group"),
and that the shuffle
+ * layer should serve this shuffle with an incremental shuffle implementation.
A plain
+ * [[ShuffleDependency]] keeps the existing semantics: its output is fully
materialized before any
+ * consumer reads it.
+ *
+ * This class only declares the capability. On its own it behaves exactly like
its parent
Review Comment:
This is no longer only a marker that behaves exactly like its parent.
Construction now routes registration to the pipelined manager and
unconditionally disables shuffle merge, so please describe those active
differences here instead of saying incremental behavior is added only by
separate components.
##########
core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala:
##########
@@ -28,12 +28,17 @@ import org.apache.spark.util.Utils
* and on each executor, based on the spark.shuffle.manager setting. The
driver registers shuffles
* with it, and executors (or tasks running locally in the driver) can ask to
read and write data.
*
+ * An implementation declares its kind by extending one of its two subtypes:
+ * [[BlockingShuffleManager]] (output is materialized as
block-manager-addressed blocks served
+ * through a [[ShuffleBlockResolver]]) or [[PipelinedShuffleManager]] (output
is read incrementally
+ * and served out-of-band).
+ *
* NOTE:
* 1. This will be instantiated by SparkEnv so its constructor can take a
SparkConf and
* boolean isDriver as parameters.
- * 2. This contains a method ShuffleBlockResolver which interacts with
External Shuffle Service
- * when it is enabled. Need to pay attention to that, if implementing a custom
ShuffleManager, to
- * make sure the custom ShuffleManager could co-exist with External Shuffle
Service.
+ * 2. A [[BlockingShuffleManager]] exposes a ShuffleBlockResolver which
interacts with the External
+ * Shuffle Service when it is enabled. Need to pay attention to that, if
implementing a custom
+ * shuffle manager, to make sure it could co-exist with the External Shuffle
Service.
Review Comment:
This requirement is currently phrased as an incomplete, conditional sentence.
```suggestion
* Shuffle Service when it is enabled. Pay attention to this when
implementing a custom shuffle
* manager to make sure it can coexist with the External Shuffle Service.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
