Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/6409
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-109820635
@srowen @vanzin This PR can cleanup correctly. I just mean without this PR
even if we add KILLED status on ApplicationMaster to check, then it can not
cleanup whe
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-109557409
@Sephiroth-Lin does it hurt to cleanup on KILLED in ApplicationMaster
though? we're trying to make more effort to clean up in this case.
---
If your project is set up fo
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-108165847
@vanzin I have tested again, and below is the result of final status when
we use yarn to kill the application:
\ | YARN UI | Driver Log | AppMaster L
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-108024459
Looks pretty good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have thi
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107997181
Ok, last patch LGTM. I wonder if it's worth it to add KILLED to the
condition here:
https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deplo
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107904636
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107904620
[Test build #33974 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33974/consoleFull)
for PR 6409 at commit
[`3a5a0a5`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107902722
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107902691
[Test build #33973 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33973/consoleFull)
for PR 6409 at commit
[`923d44d`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107854128
[Test build #33974 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33974/consoleFull)
for PR 6409 at commit
[`3a5a0a5`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107853736
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107853758
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31500453
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -91,51 +91,54 @@ private[spark] class Client(
* available in the a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107852938
[Test build #33973 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33973/consoleFull)
for PR 6409 at commit
[`923d44d`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107852550
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107852574
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107809223
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107809192
[Test build #33950 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33950/consoleFull)
for PR 6409 at commit
[`0dd7c2d`](https://git
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31491951
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -91,51 +91,54 @@ private[spark] class Client(
* available in the alpha AP
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107784995
[Test build #33950 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33950/consoleFull)
for PR 6409 at commit
[`0dd7c2d`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107784923
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107784931
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31490416
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -825,6 +813,9 @@ private[spark] class Client(
* throw an appropria
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31443005
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -825,6 +813,9 @@ private[spark] class Client(
* throw an appropriate Spar
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107428565
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107428549
[Test build #33886 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33886/consoleFull)
for PR 6409 at commit
[`b76a102`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107399589
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107399653
[Test build #33886 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33886/consoleFull)
for PR 6409 at commit
[`b76a102`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107399575
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107399469
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107302558
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107302556
**[Test build #33871 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33871/consoleFull)**
for PR 6409 at commit
[`b76a102`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107282252
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107281357
[Test build #33871 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33871/consoleFull)
for PR 6409 at commit
[`b76a102`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107280891
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107280850
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31397611
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -849,6 +852,27 @@ private[spark] class Client(
}
}
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107280587
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-107280578
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31380180
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -849,6 +852,27 @@ private[spark] class Client(
}
}
}
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106992586
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106992582
[Test build #33795 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33795/consoleFull)
for PR 6409 at commit
[`aed9f18`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106983832
[Test build #33795 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33795/consoleFull)
for PR 6409 at commit
[`aed9f18`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106983633
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106983628
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-106286738
@tgravescs yes, if yarn do it is better, but now it didn't, so as @vanzin
said may be we can do it when launcher, thank you!
---
If your project is set up for it
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105947945
> Really yarn needs a cleanup task that runs after the application master
is killed but that isn't supported at this point.
Yeah, that's too bad; in Spark we cou
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105937766
Ok, so now I remember on this. I intentionally didn't handle this case as
its really hard to be able to cleanup in this case but still not cleanup when
you want retri
Github user Sephiroth-Lin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105715873
@tgravescs I have tested below:
max retried is defaule, use yarn -kill to kill application when application
start running, run SparkPi with parameter 2.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105604628
As Tom says, this change might break application retries. I think it's time
to write a test so that people don't break that part of the code.
The test should set
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105544998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105545000
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105544965
[Test build #33514 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33514/consoleFull)
for PR 6409 at commit
[`95595c3`](https://git
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105530001
Can you please describe what happens in more detail in this case? Are you
using yarn application -kill to kill the application, does it die with out of
memory, etc. W
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105526707
Correct, you can't clean it up as the yarn RM can attempt to retry your
application (configurable number of times).
---
If your project is set up for it, you can rep
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105510514
[Test build #33514 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33514/consoleFull)
for PR 6409 at commit
[`95595c3`](https://gith
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105510292
@vanzin any reason not to clean this up on a failure? is it supposed to
stick around for retries or for debugging?
---
If your project is set up for it, you can reply to
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105510207
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6409#issuecomment-105510170
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user Sephiroth-Lin opened a pull request:
https://github.com/apache/spark/pull/6409
[SPARK-7705][Yarn] Cleanup of .sparkStaging directory fails if application
is killed
As I have tested, if we cancel or kill the app then the final status may be
undefined, killed or succeeded
61 matches
Mail list logo
