jerrypeng commented on code in PR #57286:
URL: https://github.com/apache/spark/pull/57286#discussion_r3593505162
##########
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:
##########
@@ -196,13 +195,25 @@ private[spark] class BlockManager(
// We initialize the ShuffleManager later in SparkContext and Executor, to
allow
// user jars to define custom ShuffleManagers, as such `_shuffleManager`
will be null here
- // (except for tests) and we ask for the instance from the SparkEnv.
- private lazy val shuffleManager = {
- Option(_shuffleManager).getOrElse {
- // Wait for ShuffleManager to be initialized before handling shuffle
operations.
- // Exception will be thrown if it is not initialized within the
configured timeout.
- waitForShuffleManagerInit()
- SparkEnv.get.shuffleManager
+ // (except for tests) and we ask for the resolver from the SparkEnv.
+ // The resolver used for block-by-id resolution (reads, push-merge,
decommission migration). Only
+ // a BlockingShuffleManager serves block-manager-addressed blocks; a
pipelined shuffle is served
+ // out-of-band and produces none, so these paths only ever resolve regular
shuffles. Prefer this
+ // over reaching through a ShuffleManager at the call site. Cached as a lazy
val so the
+ // init-wait and type match happen once, not on every (hot-path) block
access.
+ private lazy val shuffleBlockResolver: ShuffleBlockResolver = {
Review Comment:
Cannot replace with SparkEnv.get.shuffleBlockResolver because test inject
shuffle manager into BlockManager for testing purposes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
