mridulm commented on code in PR #36734:
URL: https://github.com/apache/spark/pull/36734#discussion_r888654676
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1885,6 +1885,16 @@ private[spark] class DAGScheduler(
mapOutputTracker.
unregisterMergeResult(shuffleId, reduceId, bmAddress,
Option(mapIndex))
}
+ } else {
+ // Unregister the merge result of <shuffleId, reduceId> if there
is a FetchFailed event
+ // and is not a MetaDataFetchException which is signified by
bmAddress being null
+ if (bmAddress != null
+ &&
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
Review Comment:
nit: Move the `&&` to the previous line.
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1885,6 +1885,16 @@ private[spark] class DAGScheduler(
mapOutputTracker.
unregisterMergeResult(shuffleId, reduceId, bmAddress,
Option(mapIndex))
}
+ } else {
+ // Unregister the merge result of <shuffleId, reduceId> if there
is a FetchFailed event
+ // and is not a MetaDataFetchException which is signified by
bmAddress being null
+ if (bmAddress != null
+ &&
bmAddress.executorId.equals(BlockManagerId.SHUFFLE_MERGER_IDENTIFIER)) {
+ assert(pushBasedShuffleEnabled, "Pushed based shuffle needs to "
+
+ "be enabled so that merge results are present.")
Review Comment:
`Pushed based shuffle` -> `Push based shuffle`
Change message to something like `Push based shuffle expected to be enabled
when handling merge block fetch failure` ? To make it clear what we are
expecting the condition to be for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
