jerrypeng commented on code in PR #57286: URL: https://github.com/apache/spark/pull/57286#discussion_r3599295429
########## core/src/main/scala/org/apache/spark/SparkEnv.scala: ########## @@ -80,12 +80,88 @@ class SparkEnv ( // user jars to define custom ShuffleManagers. @volatile private var _shuffleManager: ShuffleManager = _ + // The manager for pipelined (incrementally-readable) shuffle dependencies, configured by + // spark.shuffle.manager.incremental. Present only when that config is set. It is a peer of + // _shuffleManager, not a wrapper around it: a shuffle is routed to one or the other by its + // dependency type via `shuffleManagerFor`, so neither is ever installed "behind" the other. + @volatile private var _incrementalShuffleManager: Option[ShuffleManager] = None + // Latch to signal when the ShuffleManager has been initialized. // Used to allow callers to wait for initialization. private val shuffleManagerInitLatch = new CountDownLatch(1) + /** + * The default [[ShuffleManager]] (configured by spark.shuffle.manager), which serves all regular + * shuffle dependencies. Use this only when the intent is specifically the default manager -- e.g. + * inspecting its concrete type, or resolving a shuffle block by id (a pipelined shuffle is served + * out-of-band by the incremental manager and produces no block-manager blocks). To serve a + * specific shuffle's reads/writes, use `shuffleManagerFor`, which routes by dependency type. + */ + def defaultShuffleManager: ShuffleManager = _shuffleManager Review Comment: This is a good idea. I'll rework SparkEnv to expose managers by kind — one BlockingShuffleManager that owns all block-by-id resolution, and an optional PipelinedShuffleManager for incremental reads — with routing selecting by dependency type, and reject a config that puts a non-blocking manager in the incremental slot so there's always exactly one block-resolver source. Thanks, this is the right framing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
