Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/21577
> * t2 finishes before that kill message arrives, is allowed to commit.
> If that can happen it would generate a duplicate map output; but my guess
(hope?) is that the map output tracker would only keep one of them.
This should not happen after this PR for two reasons:
a) we do not clear status until stage finishes (which should be sufficient
to prevent the bug in entirety)
b) In (other) cases where we allow a task to commit but then kill it
(perhaps for other reasons - like user initiated kill, executor pre-emption,
etc), the task failure will be recorded and the commit state for that partition
will be cleared - and resubmitted task for partition will commit.
There is an inherent race in (b) always - where task is killed before task
completion and after output commit - that is something we cannot fix, and which
is to be handled/resolved by final job commit.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
