ulysses-you commented on code in PR #57329:
URL: https://github.com/apache/spark/pull/57329#discussion_r3655483955
##########
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:
##########
@@ -1247,6 +1318,11 @@ private[spark] class TaskSetManager(
// that the task is not running, and it is NetworkFailure rather
than TaskFailure.
case _ => !info.launching
}
+ // Grow the CPUs of the retry when a running task's executor died of a
JVM heap OOM. See
+ // the SparkOutOfMemoryError branch in handleFailedTask for the
non-fatal counterpart.
+ if (oomRetryCpusIncrement > 0 && !isBarrier && isOomExit &&
exitCausedByApp) {
Review Comment:
Thanks -- you're right that this is reachable in-tree via
`TaskMemoryManager.spillConsumer`'s double-brace map wrapped by
`enrichWriteError`, where the `NotSerializableException` fallback drops the
cause and the driver sees only the top-level `SparkException`. That gap is
about how the OOM is *thrown and transported* from the executor (a
non-serializable `SparkOutOfMemoryError` losing its cause), which is orthogonal
to this PR's scheduling change and affects the executor-exit path too. I'd
prefer to address it separately -- either making that spill-OOM serializable or
carrying an explicit serializable OOM marker on `ExceptionFailure` -- rather
than widen the scope here. Happy to file a follow-up JIRA for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
