[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210663579 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210663581 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210663327 **[Test build #55951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55951/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210620790 **[Test build #55951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55951/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210244201 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210244202 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210244107 **[Test build #55874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55874/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210212381 **[Test build #55874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55874/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210209917 jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210208056 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210208060 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210207938 **[Test build #55856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55856/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-14 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-210178429 **[Test build #55856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55856/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209712147 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209712121 **[Test build #55775 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55775/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209712142 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209710777 **[Test build #55775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55775/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209708508 Docker integration test failure likely to be unrelated, jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209707644 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209707641 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209707256 **[Test build #55763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55763/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209689400 **[Test build #55763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55763/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209663217 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209663218 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209663190 **[Test build #55747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55747/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-209660581 **[Test build #55747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55747/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-12 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-208963389 @viirya - I think having consistent accumulators only be consistent inside some of the Spark operations might be a little confusing, although that's also where I

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-12 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-208955837 I am thinking that if we can support the semantics that the accumulator updates in anyone RDD will not be recounted. Is it already enough? So we don't need to

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-12 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-208941190 +1 on "data property" instead of consistent --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r59380421 --- Diff: core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala --- @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-11 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r59269063 --- Diff: core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala --- @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r59184116 --- Diff: core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala --- @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r59183022 --- Diff: core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala --- @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r59181469 --- Diff: core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala --- @@ -0,0 +1,361 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207762628 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207762621 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207761802 **[Test build #55430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55430/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207742945 **[Test build #55430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55430/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207740012 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207739996 **[Test build #55429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55429/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207740007 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207739275 **[Test build #55429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55429/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-08 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207254301 Sounds reasonable, I'll rename things to "data property" from consistent :) Let me know if you want to chat about the API together at some point. --- If your project

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-207215712 I"m going to think about the user facing API in the next few days. For this one, can you do one change first? Have only a single flag that controls whether this is a data

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-06 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-206705518 I think with the changes so far, it might be good to change to a regular PR from WIP/RFC - any other things people think I should address before that? --- If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-206090552 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-206090548 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-206089849 **[Test build #55064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55064/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-04-05 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-206046532 **[Test build #55064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55064/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-201049661 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-201049667 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-201048896 **[Test build #54076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54076/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-200993360 **[Test build #54076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54076/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-24 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-200985173 I think the change with only wrapping the iterators with consistent accumulators present in the task should address the performance concerns @rxin had (or we can

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56422706 --- Diff: core/src/main/scala/org/apache/spark/Accumulable.scala --- @@ -114,23 +163,40 @@ class Accumulable[R, T] private ( * same mutable instance

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56417617 --- Diff: core/src/main/scala/org/apache/spark/Accumulable.scala --- @@ -53,42 +54,83 @@ import org.apache.spark.util.Utils *

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198059071 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-20 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197678803 So I did a bit of a change so we only wrap the iterators if the task involved has a consistent accumulable. --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-20 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197571739 BTW the other thing I really don't like is the current accumulator user facing API. It is way too complicated with implicit parameters etc, making it unusable in Java and

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56426605 --- Diff: core/src/main/scala/org/apache/spark/Accumulable.scala --- @@ -146,6 +212,32 @@ class Accumulable[R, T] private ( def merge(term: R) {

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197716738 **[Test build #53389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53389/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198054101 **[Test build #53462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53462/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197585737 Sorry for the confusiong. When I said "current accumulator user facing API", I literally meant the current one in Spark master branch right now. --- If your project is

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198045667 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197679180 **[Test build #53389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53389/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56421658 --- Diff: core/src/main/scala/org/apache/spark/Accumulable.scala --- @@ -146,6 +212,32 @@ class Accumulable[R, T] private ( def merge(term: R) { value_

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197582718 @squito so I don't really care about a partial partition process - but a take(1) on a cached RDD for example will fully evaluate one full partition that won't be

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56409456 --- Diff: core/src/main/scala/org/apache/spark/util/collection/Utils.scala --- @@ -36,4 +36,21 @@ private[spark] object Utils { }

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198045673 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198059041 **[Test build #53462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53462/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198039872 **[Test build #53457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53457/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56414687 --- Diff: core/src/test/scala/org/apache/spark/ConsistentAccumulatorsSuite.scala --- @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56407098 --- Diff: core/src/main/scala/org/apache/spark/util/collection/Utils.scala --- @@ -36,4 +36,21 @@ private[spark] object Utils { }

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56425894 --- Diff: core/src/main/scala/org/apache/spark/Accumulable.scala --- @@ -114,23 +163,40 @@ class Accumulable[R, T] private ( * same mutable instance

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197570550 We have recently changed accumulators so they can satisfy the use case of metrics (e.g. counting the size of bytes read) in

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56423236 --- Diff: core/src/test/scala/org/apache/spark/ConsistentAccumulatorsSuite.scala --- @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198059075 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198050832 I'm not sure why MiMa is failing on private methods but I'll add the exclusions. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197717097 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-19 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197579897 @rxin So the implicit params are the same ones used by the current regular accumulator API. So regardless of the name "consistent" - do you think having something for

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-198045638 **[Test build #53457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53457/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197585914 Oh ok yes, I agree I think the current one in master could certainly be improved - I didn't want to try and touch that at the same time as this though because its

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-19801 @rxin totally understand that this might have an unacceptable impact on performance. That part remains to be explored. For now the focus has mostly been on trying to

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r56685503 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -329,6 +334,20 @@ class TaskMetrics private[spark] (initialAccums:

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197717093 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-18 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-197550485 just thinking aloud here -- it seems like the implementation is complicated significantly by trying to support counters when you only partially read partitions, eg.

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195948561 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195948563 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195948506 **[Test build #53025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53025/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195940119 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195940102 **[Test build #53028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53028/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195940118 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195939681 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195939662 **[Test build #53029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53029/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195939680 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195926365 **[Test build #53029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53029/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195926080 **[Test build #53028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53028/consoleFull)** for PR 11105 at commit

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread holdenk
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195925790 @squito some progress on a few of the suggestions (not all for sure). Wrapping the generic compute function sounds like a good idea I'll do that next (will still need

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/11105#discussion_r55934654 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ResultTask.scala --- @@ -66,7 +67,10 @@ private[spark] class ResultTask[T, U](

[GitHub] spark pull request: [SPARK-12469][CORE][WIP/RFC] Consistent accumu...

2016-03-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11105#issuecomment-195913335 **[Test build #53025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53025/consoleFull)** for PR 11105 at commit

  1   2   >