[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-13 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/9428 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-156583956 As discussed offline, we cannot go with this approach. @zsxwing can you close this PR for now until we decide to tackle it some other way later? --- If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155265665 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155265602 **[Test build #45447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45447/consoleFull)** for PR 9428 at commit [`93c8feb`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155265666 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155256292 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155256291 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155256151 **[Test build #45435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45435/consoleFull)** for PR 9428 at commit [`49248c7`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155248415 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155248316 **[Test build #45431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45431/consoleFull)** for PR 9428 at commit [`676317b`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155245526 **[Test build #45447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45447/consoleFull)** for PR 9428 at commit [`93c8feb`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44355235 --- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala --- @@ -251,6 +252,46 @@ class CheckpointSuite extends SparkFunSuite with LocalSparkConte

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155244905 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155244937 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155228255 **[Test build #45435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45435/consoleFull)** for PR 9428 at commit [`49248c7`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155227646 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155227618 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44347259 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155224100 **[Test build #45431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45431/consoleFull)** for PR 9428 at commit [`676317b`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155223034 > Have you had a chance to test this on a real cluster? Yes. Tested this PR with Streaming `updateStateByKey`. --- If your project is set up for it, you can rep

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44344621 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44344237 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155222108 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155222125 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155195850 **[Test build #45393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45393/consoleFull)** for PR 9428 at commit [`647162f`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155196218 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155172904 @zsxwing Looks great. All my comments are pretty minor. On second thought, local checkpointing doesn't really have this problem so as long as we handle the reliable c

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44322400 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44322179 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44322134 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44321996 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -258,11 +258,14 @@ abstract class RDD[T: ClassTag]( * subclasses of RDD.

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44321584 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -51,6 +65,27 @@ private[spark] class ReliableRDDCheckpointData[T:

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44321308 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -258,11 +258,14 @@ abstract class RDD[T: ClassTag]( * subclasses of RDD.

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44321310 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320866 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -63,11 +98,20 @@ private[spark] class ReliableRDDCheckpointData[T

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320738 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -63,11 +98,20 @@ private[spark] class ReliableRDDCheckpointData[T

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320580 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320462 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320395 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320330 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320223 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44320118 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44319893 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44319855 --- Diff: core/src/main/scala/org/apache/spark/util/CheckpointingIterator.scala --- @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r44315469 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -31,12 +31,26 @@ import org.apache.spark.util.SerializableConfigu

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155156399 **[Test build #45393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45393/consoleFull)** for PR 9428 at commit [`647162f`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155153659 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155153687 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-09 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-155153532 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-03 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153546887 /cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-03 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r43798587 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -31,12 +31,26 @@ import org.apache.spark.util.SerializableConfigurat

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153232648 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153232651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153232583 **[Test build #44863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44863/consoleFull)** for PR 9428 at commit [`647162f`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread tedyu
Github user tedyu commented on a diff in the pull request: https://github.com/apache/spark/pull/9428#discussion_r43711427 --- Diff: core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala --- @@ -31,12 +31,26 @@ import org.apache.spark.util.SerializableConfiguratio

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153214364 **[Test build #44863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44863/consoleFull)** for PR 9428 at commit [`647162f`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153213809 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153213792 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/9258 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9428#issuecomment-153213596 /cc @andrewor14 @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-11-02 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-153213531 @andrewor14 I just opened #9428 to take over #7021 and fix potential issues in the previous PR. --- If your project is set up for it, you can reply to this email and ha

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152178362 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152178361 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152178180 **[Test build #44590 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44590/consoleFull)** for PR 9258 at commit [`824be91`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152154426 @zsxwing I took a quick look and I have a high level question. Why not just do the checkpointing iterator? IIUC this approach involves reading the iterator back from

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152152727 **[Test build #44590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44590/consoleFull)** for PR 9258 at commit [`824be91`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152152043 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152151993 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-29 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-152151854 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151507554 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151507553 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151507411 **[Test build #44429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44429/consoleFull)** for PR 9258 at commit [`824be91`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151472511 There is still one problem in this PR. Currently, if an RDD is not persisted, this RDD will be recomputed to generate an Iterator after checkpointing (We cannot use the

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151472268 **[Test build #44429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44429/consoleFull)** for PR 9258 at commit [`824be91`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151470527 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151470440 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151470042 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151455066 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151455069 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151454952 **[Test build #44417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44417/consoleFull)** for PR 9258 at commit [`824be91`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151420887 **[Test build #44417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44417/consoleFull)** for PR 9258 at commit [`824be91`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151418311 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151418271 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151231402 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151231407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151231061 **[Test build #44360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44360/consoleFull)** for PR 9258 at commit [`d69f775`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151196144 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151195951 **[Test build #44355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44355/consoleFull)** for PR 9258 at commit [`8ae42e0`](https://git

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151196146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151188331 **[Test build #44360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44360/consoleFull)** for PR 9258 at commit [`d69f775`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151182876 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151182814 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151154443 **[Test build #44355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44355/consoleFull)** for PR 9258 at commit [`8ae42e0`](https://gith

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151149704 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151149827 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-25 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-151023896 @andrewor14 Can you please take a look at this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-25 Thread ryan-williams
Github user ryan-williams commented on a diff in the pull request: https://github.com/apache/spark/pull/9258#discussion_r42951830 --- Diff: core/src/main/scala/org/apache/spark/CheckpointManager.scala --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-25 Thread ryan-williams
Github user ryan-williams commented on a diff in the pull request: https://github.com/apache/spark/pull/9258#discussion_r42951772 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -81,6 +81,8 @@ class SparkEnv ( @deprecated("Actor system is no longer suppor

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-25 Thread ryan-williams
Github user ryan-williams commented on a diff in the pull request: https://github.com/apache/spark/pull/9258#discussion_r42951707 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -258,6 +258,16 @@ abstract class RDD[T: ClassTag]( * subclasses of RDD.

[GitHub] spark pull request: [SPARK-8582][Core]Optimize checkpointing to av...

2015-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9258#issuecomment-150746506 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

  1   2   >