GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/21607
branch-2.1: backport SPARK-24589 and SPARK-22897
*Please do not merge this PR directly.*
I'm posting this to speed up testing and review. These should go in as two
separate commits. I'll address any test failures and review feedback here, but
will push the changes separately when everything is in good shape.
See individual commits for each individual change.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-24589-2.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21607
----
commit e5ccac21db69a5698e70d8fb993296fa854de132
Author: Xianjin YE <advancedxy@...>
Date: 2018-01-02T15:30:38Z
[SPARK-22897][CORE] Expose stageAttemptId in TaskContext
stageAttemptId added in TaskContext and corresponding construction
modification
Added a new test in TaskContextSuite, two cases are tested:
1. Normal case without failure
2. Exception case with resubmitted stages
Link to [SPARK-22897](https://issues.apache.org/jira/browse/SPARK-22897)
Author: Xianjin YE <advance...@gmail.com>
Closes #20082 from advancedxy/SPARK-22897.
(cherry picked from commit a6fc300e91273230e7134ac6db95ccb4436c6f8f)
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
commit 034fe94d544d5235eac70e38c7891c7110dbb569
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-06-21T18:25:15Z
[SPARK-24589][CORE] Correctly identify tasks in output commit coordinator
[branch-2.1].
When an output stage is retried, it's possible that tasks from the previous
attempt are still running. In that case, there would be a new task for the
same partition in the new attempt, and the coordinator would allow both
tasks to commit their output since it did not keep track of stage attempts.
The change adds more information to the stage state tracked by the
coordinator,
so that only one task is allowed to commit the output in the above case.
The stage state in the coordinator is also maintained across stage retries,
so that a stray speculative task from a previous stage attempt is not
allowed
to commit.
This also removes some code added in SPARK-18113 that allowed for duplicate
commit requests; with the RPC code used in Spark 2, that situation cannot
happen, so there is no need to handle it.
Author: Marcelo Vanzin <van...@cloudera.com>
Closes #21577 from vanzin/SPARK-24552.
(cherry picked from commit c8e909cd498b67b121fa920ceee7631c652dac38)
Signed-off-by: Thomas Graves <tgra...@apache.org>
(cherry picked from commit 751b008204a847e26d79b1996ce3f3dbe96a5acf)
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
