dongjoon-hyun opened a new pull request, #59792: URL: https://github.com/apache/spark/pull/59792
### What changes were proposed in this pull request? This PR guards the `markTaskFailed(e)` call in `TaskContextImpl.markInterrupted`. If a task failure listener throws while the failure of the interrupt listeners is being recorded, the resulting `TaskCompletionListenerException` is no longer propagated. It is attached to the recorded task failure as a suppressed exception instead. Only `TaskCompletionListenerException` is caught, so fatal errors still propagate. ### Why are the changes needed? SPARK-56330 made `markInterrupted` swallow the exceptions of failing `TaskInterruptListener`s and record them with `markTaskFailed(e)`. That call runs the task failure listeners, and `invokeListeners` throws a new `TaskCompletionListenerException` when one of them fails. The exception escapes `markInterrupted`: - `Task.kill` calls `context.markInterrupted(reason)` before `taskThread.interrupt()`, so the interrupt is skipped and a task thread blocked in an interruptible wait or I/O is never woken. - With `spark.task.reaper.enabled=true`, the kill runs inside `TaskReaper.run`, where the exception reaches the uncaught exception handler and the executor JVM exits. Without the reaper, the RPC `Inbox` only logs "Ignoring error". - `createResourceUninterruptibly` calls `markInterrupted` the same way and skips its own `Thread.interrupt()`. - It contradicts the documentation of `TaskContext.addTaskInterruptListener`, which says exceptions are not propagated from `markInterrupted`. The failure is already recorded by `markTaskFailed` before the listeners run, and each failing listener is already logged by `invokeListeners`, so nothing is lost by not propagating it. ### Does this PR introduce _any_ user-facing change? No API or configuration change. In the rare case where an interrupt listener and a task failure listener both throw, killing a task now interrupts the task thread as documented, instead of failing with `TaskCompletionListenerException` (and, with the task reaper enabled, exiting the executor JVM). This behavior exists in released Spark 4.2.0. ### How was this patch tested? Added two tests to `TaskContextSuite`. Both fail without the fix with `TaskCompletionListenerException: exception in failure listener` thrown from `markInterrupted` (and from `Task.kill`), and pass with it: - `SPARK-60554: a failing TaskFailureListener does not escape markInterrupted` - `SPARK-60554: Task.kill interrupts the task thread even if interrupt and failure listeners fail` ``` build/sbt 'core/testOnly *TaskContextSuite *BarrierTaskContextSuite *ExecutorSuite *TaskSetManagerSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
