Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1056
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50915278
Looks good Sandy, I'm going to merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50870753
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50867006
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17669/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50866851
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50865776
QA results for PR 1056:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50861872
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17668/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50861360
QA results for PR 1056:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50861124
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17667/consoleFull
---
If
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50859536
Yeah I think it's fine to put that too another patch and just make it
something a bit more on the conservative side (10 seconds) for now.
---
If your project is set up
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15685394
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -320,6 +323,26 @@ private[spark] class TaskSchedulerImpl(
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15684813
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -991,6 +994,9 @@ class SparkContext(config: SparkConf) extends Logging {
dagSch
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50853874
Thanks @pwendell and @andrewor14 for your continued reviews.
10 seconds sounds fine to me. Not that it's a shining beacon of
performance, but MapReduce actually us
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50850298
QA results for PR 1056:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50849206
Sandy - I took a pass on this. Mostly minor comments, but I did propose
lowering the default frequency from 2 seconds. Overall this is looking in good
shape.
---
If yo
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15682407
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -171,28 +155,70 @@ class JobProgressListener(conf: SparkConf) extends
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50849017
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17646/consoleFull
---
If
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15682370
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -56,7 +56,7 @@ private[jobs] object UIData {
}
case class TaskU
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15682364
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -56,7 +56,7 @@ private[jobs] object UIData {
}
case class TaskU
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50848855
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50848434
QA results for PR 1056:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15682191
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -350,4 +353,47 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15682056
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -320,6 +323,26 @@ private[spark] class TaskSchedulerImpl(
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15681944
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -38,8 +37,10 @@ import org.apache.spark._
import org.apache.spark.exe
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15681898
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -991,6 +994,9 @@ class SparkContext(config: SparkConf) extends Logging {
dag
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50847355
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17641/consoleFull
---
If
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15681834
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -50,6 +50,7 @@ import org.apache.spark.scheduler.local.LocalBackend
import org.a
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50847179
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50560229
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50556357
Latest patch incorporates latest feedback and adds BlockManagerSuite back
in. I tested on a small cluster and saw executors shut down fine (but haven't
run at scale).
--
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50556346
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17393/consoleFull
---
If
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15560393
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15559948
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15559871
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
}
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15543587
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -38,8 +37,10 @@ import org.apache.spark._
import org.apache.spark.
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15542781
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15538656
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15528611
--- Diff: docs/configuration.md ---
@@ -524,6 +524,13 @@ Apart from these, the following properties are also
available, and may be useful
output direct
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15527486
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -56,7 +56,7 @@ private[jobs] object UIData {
}
case class TaskUIDa
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15526958
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -155,6 +156,23 @@ class DAGScheduler(
eventProcessActor ! Completion
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15526822
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
}
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50435407
Hey @sryza I think the overall architecture here is good, but I did a pass
with various comments. I do have a few questions throughout that are about
nontrivial things,
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506372
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -1,5 +1,4 @@
/*
- * Licensed to the Apache Software Foundation
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506371
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ---
@@ -341,24 +339,6 @@ class BlockManagerSuite extends FunSuite with Matcher
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506267
--- Diff: docs/configuration.md ---
@@ -524,6 +524,13 @@ Apart from these, the following properties are also
available, and may be useful
output dir
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506235
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -155,6 +156,23 @@ class DAGScheduler(
eventProcessActor ! Complet
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506106
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -23,6 +23,14 @@ import org.apache.spark.storage.{BlockId, BlockStatus}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506095
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506065
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -48,6 +48,8 @@ private[spark] class Executor(
private val EMPTY_BYT
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506056
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506045
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15506011
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -186,6 +191,7 @@ private[spark] class Executor(
// Run the ac
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15505852
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -186,6 +191,7 @@ private[spark] class Executor(
// Run the ac
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15505792
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -186,6 +191,7 @@ private[spark] class Executor(
// Run the ac
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15496699
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -155,6 +156,23 @@ class DAGScheduler(
eventProcessActor ! Complet
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r1549
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15493992
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -155,6 +156,23 @@ class DAGScheduler(
eventProcessActor ! Complet
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15493527
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -56,7 +56,7 @@ private[jobs] object UIData {
}
case class TaskU
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15492760
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +353,48 @@ private[spark] class Executor(
}
}
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50179286
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50173304
As far as I can tell, you're right - I don't see why updateShuffleMetrics
needs to be synchronized.
Uploading a patch that:
* Adds comments to TaskMetrics that
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-50173793
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17186/consoleFull
---
If
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15333186
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +352,46 @@ private[spark] class Executor(
}
}
}
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49975385
I was actually a bit confused why `updateShuffleReadMetrics` is
synchronized. Can that be called from multiple threads as-is? I wasn't aware of
cases where we had multi-
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49970751
I don't entirely understand the advantage of having a separate
PartialTaskMetrics. Ultimately every field of TaskMetrics except for maybe
shuffleFinishTime will be able to
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49963097
Hey @sryza so I spent a good amount of time talking about this with
@aarondav and @andrewor14.
There are some challenges because we break assumptions in the prev
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15273504
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +352,46 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15273373
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +352,46 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15273232
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -348,4 +352,46 @@ private[spark] class Executor(
}
}
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15272543
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15272172
--- Diff: docs/configuration.md ---
@@ -502,6 +502,13 @@ Apart from these, the following properties are also
available, and may be useful
output dir
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49821798
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49815131
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16998/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-4981
QA results for PR 1056:- This patch FAILED unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class SparkListenerExe
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49814034
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16996/consoleFull
---
If
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49813666
Made stylistic fixes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49805158
I ran some tests using SizeEstimator and the sizes seem pretty reasonable
to me.
A heartbeat with no tasks running is 232 bytes.
A heartbeat with two tasks runn
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49785120
Hey @sryza, I left a bunch of minor comments, but other than that this
looks pretty in shape. We just need to actually test how much overhead this
adds to see if we ne
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248346
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -56,7 +56,7 @@ private[jobs] object UIData {
}
case class Tas
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248363
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -124,4 +127,59 @@ private[spark] object AkkaUtils extends Logging {
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248247
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -171,28 +155,67 @@ class JobProgressListener(conf: SparkConf) exten
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248220
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -113,7 +113,7 @@ class JobProgressListener(conf: SparkConf) extends
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248135
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -171,28 +155,67 @@ class JobProgressListener(conf: SparkConf) exten
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15248103
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -171,28 +155,67 @@ class JobProgressListener(conf: SparkConf) exten
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247748
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
---
@@ -21,6 +21,8 @@ import java.nio.ByteBuffer
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247722
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala ---
@@ -54,4 +56,8 @@ private[spark] trait TaskScheduler {
// Get
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247764
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -320,6 +323,24 @@ private[spark] class TaskSchedulerImpl(
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247713
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -320,6 +323,24 @@ private[spark] class TaskSchedulerImpl(
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247680
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -75,6 +75,9 @@ case class
SparkListenerBlockManagerRemoved(blockManage
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247591
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -155,6 +156,21 @@ class DAGScheduler(
eventProcessActor ! Compl
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15247437
--- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49707887
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class Heartbeat(class
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49707381
QA results for PR 1056:- This patch PASSES unit tests.- This patch
merges cleanly- This patch adds the following public classes
(experimental):case class Heartbeat(class
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49701430
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16945/consoleFull
---
If
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49700944
QA tests have started for PR 1056. This patch merges cleanly. View
progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16944/consoleFull
---
If
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49700737
Upmerged and incorporated review comments.
Also added a random sleep at the start so that the executor heartbeats are
less likely to get in sync.
---
If your proj
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1056#issuecomment-49684190
@sryza I don't know of a correct way to check the size of your akka
message, but one quick way to find out approximately how large your messages
are is use `SizeEstima
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15204465
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -154,6 +160,16 @@ class DAGScheduler(
eventProcessActor ! Compl
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15204407
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -129,7 +128,7 @@ class BlockManagerMasterActor(val isLocal: Boolean,
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1056#discussion_r15203706
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -154,6 +160,16 @@ class DAGScheduler(
eventProcessActor ! Compl
1 - 100 of 177 matches
Mail list logo
