[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread alexrovner
Github user alexrovner commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97866642 Changes merged upstream. Closing pull. https://github.com/apache/spark/commit/ec196ab1c7569d7ab0a50c9d7338c2835f2c84d5 --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread alexrovner
Github user alexrovner closed the pull request at: https://github.com/apache/spark/pull/5793 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97734390 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97734392 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/314

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97734372 [Test build #31409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31409/consoleFull) for PR 5793 at commit [`f238f94`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97710144 [Test build #31409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31409/consoleFull) for PR 5793 at commit [`f238f94`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97710043 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97710020 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97709876 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabl

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-30 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97709893 As a cherry pick LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5793#issuecomment-97608996 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-04-29 Thread alexrovner
GitHub user alexrovner opened a pull request: https://github.com/apache/spark/pull/5793 [SPARK-5529][CORE]Add expireDeadHosts in HeartbeatReceiver If a blockManager has not send heartBeat more than 120s, BlockManagerMasterActor will remove it. But coarseGrainedSchedulerBackend can

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4363 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76321168 Great, LGTM thanks for your work @shenh062326 I'm merging this into master. --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76320309 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76320304 [Test build #28030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28030/consoleFull) for PR 4363 at commit [`2c9a46a`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76312402 [Test build #28030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28030/consoleFull) for PR 4363 at commit [`2c9a46a`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25464081 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25464024 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1051,7 +1058,7 @@ class SparkContext(config: SparkConf) extends Logging with Exec

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25463974 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25463799 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25463943 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76161857 [Test build #27997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27997/consoleFull) for PR 4363 at commit [`1a042ff`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76161867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76151469 [Test build #27997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27997/consoleFull) for PR 4363 at commit [`1a042ff`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146314 [Test build #27993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27993/consoleFull) for PR 4363 at commit [`2dc456e`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146315 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76146091 [Test build #27993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27993/consoleFull) for PR 4363 at commit [`2dc456e`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-76145260 Sorry for late, I will change it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-26 Thread shenh062326
Github user shenh062326 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25413198 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-24 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25312597 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-24 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25282637 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25095951 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25094287 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25086792 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25086274 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25085897 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-20 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25084604 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043997 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-75175379 @shenh062326 Thanks for addressing the feedback. I think the current solution works, and the comments I left inline are mostly minor. I will merge this once you addres

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043904 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043871 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043678 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043444 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043138 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala --- @@ -73,5 +73,9 @@ private[spark] trait TaskScheduler { * @return A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25043098 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042970 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1035,6 +1035,8 @@ class SparkContext(config: SparkConf) extends Logging with Exec

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042704 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042663 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042565 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042387 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r25042286 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74206518 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74206515 [Test build #27416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27416/consoleFull) for PR 4363 at commit [`d221493`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74201738 [Test build #27416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27416/consoleFull) for PR 4363 at commit [`d221493`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74059407 [Test build #27350 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27350/consoleFull) for PR 4363 at commit [`7448ac6`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74059408 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74059184 [Test build #27350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27350/consoleFull) for PR 4363 at commit [`7448ac6`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-12 Thread shenh062326
Github user shenh062326 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24574765 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-11 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74029575 Thanks for updating this @shenh062326. I left a few additional comments. It would probably be good for @andrewor14 to take a look as well. --- If your project is set up

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-11 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24563260 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-11 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24563184 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1034,6 +1034,14 @@ class SparkContext(config: SparkConf) extends Logging with Executor

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-11 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24563093 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,84 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-11 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-74023636 Hi @sryza, I think this pull request is OK now, can you merge it into master? --- If your project is set up for it, you can reply to this email and have your reply a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73644896 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73644891 [Test build #27167 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27167/consoleFull) for PR 4363 at commit [`b904aed`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73639938 [Test build #27167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27167/consoleFull) for PR 4363 at commit [`b904aed`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73638561 The failed tests have no relationship with this patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73634364 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73634355 [Test build #27156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27156/consoleFull) for PR 4363 at commit [`52725af`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73633932 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73633923 [Test build #27155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27155/consoleFull) for PR 4363 at commit [`5bedcb8`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73629089 [Test build #27156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27156/consoleFull) for PR 4363 at commit [`52725af`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73628553 [Test build #27155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27155/consoleFull) for PR 4363 at commit [`5bedcb8`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73626671 @andrewor14 @rxin yes, i agree with you. for other mode, later we need to implement killing executor. so this PR is unify failure detection between blockmanager an

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread shenh062326
Github user shenh062326 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24381524 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,82 @@ package org.apache.spark -import akka.a

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73625458 Hi @andrewor14 , @sryza and @rxin. Thanks. I agree with your views. I will change sc.killExecutor to not throw an assertion error. --- If your project is set up for

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73626450 Hey @shenh062326 actually a better thing to do here might be to just check whether the `sc` is running YARN mode, and call `sc.killExecutor` only if that's the case, m

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73615590 Ok, per @rxin's offline suggestion maybe it still makes sense to get this fix in even just for YARN. Though in this patch we'll have to change `sc.killExecutor` so tha

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73615286 I think it is fairly important to unify failure detection, i.e. block managers and executors should fate share. Having two different code paths for failure detection is boun

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73614405 @andrewor14 do you think it's worthwhile to make a best effort attempt to kill executors on cluster managers that support it? I know `killExecutors` doesn't currently work

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73610995 @shenh062326 Thanks for your work. However, the approach here won't actually work for standalone or Mesos mode. In general once the executor is launched the driver doe

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24374228 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,82 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24374107 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,82 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24374036 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,82 @@ package org.apache.spark -import akka.ac

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-08 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73455076 Thanks for walking me through the chain of calls. I assumed there was documentation for the conf properties we're replacing, but it seems like there's none. I'm OK

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73229173 [Test build #26913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26913/consoleFull) for PR 4363 at commit [`a858fb5`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73229181 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73221171 [Test build #26913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26913/consoleFull) for PR 4363 at commit [`a858fb5`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread shenh062326
Github user shenh062326 commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73220183 scheduler.executorLost(executorId, SlaveLost()) will call BlockManagerMasterActor.removeBlockManager, the stack is: HeartbeatReceiver.expireDeadHosts TaskSched

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73207417 This is looking close. Had a few more nits and a couple larger comments: * The documentation needs to be updated. I'm happy to provide wording if it would be helpful.

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24229237 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24229250 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-06 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4363#discussion_r24229185 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -17,33 +17,86 @@ package org.apache.spark -import akka.actor.A

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73174253 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73174249 [Test build #26880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26880/consoleFull) for PR 4363 at commit [`07952f3`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73173512 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73173505 [Test build #26881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26881/consoleFull) for PR 4363 at commit [`6bab7aa`](https://gith

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73166793 [Test build #26881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26881/consoleFull) for PR 4363 at commit [`6bab7aa`](https://githu

[GitHub] spark pull request: [SPARK-5529][CORE]Add expireDeadHosts in Heart...

2015-02-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4363#issuecomment-73165883 [Test build #26880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26880/consoleFull) for PR 4363 at commit [`07952f3`](https://githu

  1   2   >