[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-06-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5233 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-06-29 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-116884073 OK, I have closed SPARK-4069. @PraveenSeluka would you mind closing this patch? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-06-29 Thread PraveenSeluka
Github user PraveenSeluka commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-116881567 @sryza - After the fix on YARN side, I have not seen this issue - We can close this (and also the JIRA). --- If your project is set up for it, you can reply to t

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-06-29 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-116877465 There's no bug on the Spark side. @PraveenSeluka is this still something you're running into? If this is causing pain to a lot of users it could be worth the workaround,

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-06-29 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-116847131 Looks like YARN-3415 is resolved. Is there still a fix that's needed on the Spark side @sryza? If not, shall we close this PR? --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-28 Thread PraveenSeluka
Github user PraveenSeluka commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87270665 @sryza - Yeah, its not really a Spark issue. Do you think, we can get this fix in on spark side too ? --- If your project is set up for it, you can reply to this e

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-28 Thread mindprince
Github user mindprince commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87185350 There wasn't - I created https://issues.apache.org/jira/browse/YARN-3415 just now. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87165635 Thanks for the explanation @mindprince. That makes sense. This really seems like a bug on the YARN side though - we're counting containers as AMs that are not AMs. A fix

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-27 Thread mindprince
Github user mindprince commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87160289 Hi @sryza We faced an issue where not doing this was causing incorrect accounting of the queue's AM Resources. As you may be aware that the YARN fair

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87103917 Hi @PraveenSeluka, what's the disadvantage of just letting YARN handle this? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5233#issuecomment-87068281 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-4069] [YARN] When AppMaster finishes, t...

2015-03-27 Thread PraveenSeluka
GitHub user PraveenSeluka opened a pull request: https://github.com/apache/spark/pull/5233 [SPARK-4069] [YARN] When AppMaster finishes, this change calls AMRMClient to removeContainer... Currently Application Master goes down without releasing the containers. Once AM goes down, YAR