[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153414851 Thanks Marcelo! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8093 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153414205 Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153404861 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153404862 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153404652 **[Test build #44919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44919/consoleFull)** for PR 8093 at commit [`d0a2f44`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153401864 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153401863 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153401701 **[Test build #44923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44923/consoleFull)** for PR 8093 at commit [`4acf2e9`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153366300 **[Test build #44923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44923/consoleFull)** for PR 8093 at commit [`4acf2e9`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153365863 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153365937 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153357278 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153357269 **[Test build #44921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44921/consoleFull)** for PR 8093 at commit [`90b2b90`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153357280 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153356058 **[Test build #44921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44921/consoleFull)** for PR 8093 at commit [`90b2b90`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153355818 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153355848 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153352450 **[Test build #44919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44919/consoleFull)** for PR 8093 at commit [`d0a2f44`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153350280 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-153350234 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-31 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-152786751 LGTM but needs a rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r43578203 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -189,7 +188,9 @@ class CoarseGrainedSchedulerBacke

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151337484 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151337268 **[Test build #44383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44383/consoleFull)** for PR 8093 at commit [`4b4e5dd`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151337488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151314454 **[Test build #44383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44383/consoleFull)** for PR 8093 at commit [`4b4e5dd`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151313105 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151313091 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151201804 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151201802 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151201793 **[Test build #44362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44362/consoleFull)** for PR 8093 at commit [`91fa0f9`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151200883 **[Test build #44362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44362/consoleFull)** for PR 8093 at commit [`91fa0f9`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r43016585 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -367,9 +367,15 @@ private[spark] object JsonProtocol { ("Job ID"

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151198182 Thanks @vanzin for your feedback. I have made your suggested changes, build/tested again and updated this pull request. Thanks again! --- If your project is set up f

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151196890 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-151196859 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150869715 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150869716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150869698 **[Test build #44301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44301/consoleFull)** for PR 8093 at commit [`5f99397`](https://git

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150852484 **[Test build #44301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44301/consoleFull)** for PR 8093 at commit [`5f99397`](https://gith

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150852007 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150852000 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-24 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150851790 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150698699 Looks OK, just a few style nits remaining. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42916821 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -217,11 +217,14 @@ case class TaskCommitDenied( * the task crashed the JVM.

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42916732 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -367,9 +367,15 @@ private[spark] object JsonProtocol { ("Job ID" -> t

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42916428 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -367,9 +367,15 @@ private[spark] object JsonProtocol { ("Job ID" -> t

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42916134 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -189,7 +188,8 @@ class CoarseGrainedSchedulerBacke

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42915834 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -217,11 +217,14 @@ case class TaskCommitDenied( * the task crashed the JVM.

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-23 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r42915637 --- Diff: core/src/main/scala/org/apache/spark/TaskEndReason.scala --- @@ -217,11 +217,14 @@ case class TaskCommitDenied( * the task crashed the JVM.

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-10-22 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-150401167 Hi all, thanks for your feedback. I have updated this branch now, ran the unit-tests which are green. The previously much talked about race condition has been taken c

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-18 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-132291876 Thanks Marcelo, the code more or less works but GetExecutorLossReason is getting called twice. So, I need to work that out with @mccheah. Once that's resolved, I will

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-17 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-131947059 @markgrover you need to update your branch to fix conflicts, after that we can trigger tests for this. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-14 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-131185171 I am have ported (aka stolen) some relevant ideas from #8007 here, just pushed another commit. I still have some concerns regarding the testing thereof, so I am going

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-13 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130865695 Thanks @vanzin. I looked at #8007 and some pieces of it are definitely very relevant. So, I will use some of the ideas from there. In full disclosure, I did d

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-13 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130803328 I haven't looked closely at the code yet, but #8007 probably needs to fix a similar issue to the race you have here; might be a source of ideas. --- If your project is s

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-12 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130408846 Updated the review incorporating all the review feedback. In the meanwhile, let me poke a little more into the race condition and get back with what I find. --- If y

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-12 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130385210 I chatted w/ Mark about this a bit. I *think* both calls will get to [`removeExecutor` in `BlockManagerMasterEndpoint`](https://github.com/apache/spark/blob/be5d1912076c

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130167519 > guess we could do that. My concern is that the race is probably always going to be won by the executor disconnect message (instead of the explicit RemoveExecutor m

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130159464 Thanks for the review, @squito. I did consider that as well. The UI listens to whatever is sent as `SparkListenerTaskEnd` message by the DAGScheduler::handleTaskCompl

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36825910 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -435,19 +437,24 @@ private[yarn] class YarnAllocator( //

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36825911 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -423,6 +424,7 @@ private[yarn] class YarnAllocator( for (com

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130137795 > If you only have one event, then just do the best you can with that one. It's a little unexpected to receive two "removed" events for a single "added" one. It'

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130133128 Just throwing an idea out on dealing with the race for getting the full reason (without fully understanding the details yet myself): couldn't this just be handled at the

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36819713 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -435,19 +437,24 @@ private[yarn] class YarnAllocator( // the

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36819662 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -423,6 +424,7 @@ private[yarn] class YarnAllocator( for (complet

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130103072 > there are cases where the RemoveExecutor event may never come, right That can happen, yes. In a normal RPC system, that message not arriving would probably mea

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130084493 @andrewor14 [here](https://issues.apache.org/jira/secure/attachment/12749771/error_showing_in_UI.png)'s the link from the JIRA. --- If your project is set up for it

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130082220 @markgrover can you post a screenshot? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36794461 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,8 @@ private[spark] object JsonProtocol { case `taskKil

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36793927 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,8 @@ private[spark] object JsonProtocol { case `taskKilled`

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130053497 Thanks for the review, @vanzin! Indeed, the race is tricky. Yup, I like that solution but there are cases where the RemoveExecutor event may never come, right (in the

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36793126 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,8 @@ private[spark] object JsonProtocol { case `taskKil

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36792140 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -110,6 +114,7 @@ private[spark] abstract class YarnSche

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36784216 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -110,6 +114,7 @@ private[spark] abstract class YarnSchedule

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-130016534 The race part is pretty tricky. The only solution I can think of is to delay the posting of the `SparkListenerExecutorRemoved` event until an explicit `RemoveExecutor` me

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36783484 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -423,6 +424,7 @@ private[yarn] class YarnAllocator( for (complet

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36783459 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -423,6 +424,7 @@ private[yarn] class YarnAllocator( for (complet

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36783358 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -110,6 +114,7 @@ private[spark] abstract class YarnSche

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36783267 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,8 @@ private[spark] object JsonProtocol { case `taskKilled`

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782853 --- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala --- @@ -795,7 +796,8 @@ private[spark] object JsonProtocol { case `taskKilled`

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782545 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -17,6 +17,10 @@ package org.apache.spark.sch

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread markgrover
Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782462 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -19,6 +19,9 @@ package org.apache.spark.schedu

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782349 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -110,6 +114,7 @@ private[spark] abstract class YarnSchedule

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782307 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -17,6 +17,10 @@ package org.apache.spark.schedul

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-11 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36782151 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -19,6 +19,9 @@ package org.apache.spark.scheduler.

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-10 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-129712836 There's one thing I'd really appreciate people's thoughts on: There seems to be a race condition related to displaying the error message in the UI. When YARN

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-10 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-129710600 Jenkins, please test this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-10 Thread markgrover
Github user markgrover commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-129709754 This pull request is meant to achieve two goals: 1. Show in driver logs, primarily in yarn client mode, if YARN is killing containers because of one or more of the

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8093#issuecomment-129706966 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: SPARK-9790: [YARN] Expose in WebUI if NodeMana...

2015-08-10 Thread markgrover
GitHub user markgrover opened a pull request: https://github.com/apache/spark/pull/8093 SPARK-9790: [YARN] Expose in WebUI if NodeManager is the reason why executors were killed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/m