venkata91 commented on code in PR #58437:
URL: https://github.com/apache/spark/pull/58437#discussion_r3997004243
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -4334,7 +4344,9 @@ private[spark] class DAGScheduler(
message: String): Unit = {
logInfo(log"Shuffle files lost for worker ${MDC(WORKER_ID, workerId)} " +
log"on host ${MDC(HOST, host)}")
- mapOutputTracker.removeOutputsOnHost(host)
+ // Worker loss (not a fetch failure): reliably-stored shuffle output lives
off the worker and
Review Comment:
Done. Qualified the Scaladoc to say only non-reliably-stored output is
treated as lost on executor/worker loss, and added `@param` entries for
`ignoreShuffleFileLostEpoch` and `respectReliablyStored`.
##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -4267,7 +4272,8 @@ private[spark] class DAGScheduler(
fileLost: Boolean,
hostToUnregisterOutputs: Option[String],
maybeEpoch: Option[Long] = None,
- ignoreShuffleFileLostEpoch: Boolean = false): Unit = {
+ ignoreShuffleFileLostEpoch: Boolean = false,
Review Comment:
Done. Qualified the Scaladoc to say only non-reliably-stored output is
treated as lost on executor/worker loss, and added `@param` entries for
`ignoreShuffleFileLostEpoch` and `respectReliablyStored`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
