jerrypeng commented on code in PR #57286:
URL: https://github.com/apache/spark/pull/57286#discussion_r3599369585
##########
core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala:
##########
@@ -28,14 +28,19 @@ import org.apache.spark.util.Utils
* and on each executor, based on the spark.shuffle.manager setting. The
driver registers shuffles
* with it, and executors (or tasks running locally in the driver) can ask to
read and write data.
*
+ * This trait is sealed and is not meant to be implemented directly. A shuffle
manager declares its
+ * kind by extending one of its two subtypes: [[BlockingShuffleManager]]
(output is materialized as
+ * block-manager-addressed blocks served through a [[ShuffleBlockResolver]]) or
+ * [[PipelinedShuffleManager]] (output is read incrementally and served
out-of-band).
+ *
* NOTE:
* 1. This will be instantiated by SparkEnv so its constructor can take a
SparkConf and
* boolean isDriver as parameters.
- * 2. This contains a method ShuffleBlockResolver which interacts with
External Shuffle Service
- * when it is enabled. Need to pay attention to that, if implementing a custom
ShuffleManager, to
- * make sure the custom ShuffleManager could co-exist with External Shuffle
Service.
+ * 2. A [[BlockingShuffleManager]] exposes a ShuffleBlockResolver which
interacts with the External
+ * Shuffle Service when it is enabled. Need to pay attention to that, if
implementing a custom
+ * shuffle manager, to make sure it could co-exist with the External Shuffle
Service.
*/
-private[spark] trait ShuffleManager {
+private[spark] sealed trait ShuffleManager {
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
