[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk closed the pull request at: https://github.com/apache/spark/pull/10841 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180844069 I'm going to close this and re-open one with the "full auto" type approach suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180721409 **[Test build #50868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50868/consoleFull)** for PR 10841 at commit [`5a7bd6f`](https://github.com/apache/spark/commit/5a7bd6fad9cb2bfb054d1377841d641a6c5adb27). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180733770 **[Test build #50868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50868/consoleFull)** for PR 10841 at commit [`5a7bd6f`](https://github.com/apache/spark/commit/5a7bd6fad9cb2bfb054d1377841d641a6c5adb27). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180733841 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180733844 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50868/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180540823 @andrewor14 I changed it around a bit so we don't need to have the user explicitly pass the rdd/partition information to the consistent accumulator. It's still got the separate class since we need to keep track of some extra information for consistent accumulators. Think this is closer to the correct direction? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180554724 **[Test build #50842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50842/consoleFull)** for PR 10841 at commit [`91c4f0e`](https://github.com/apache/spark/commit/91c4f0ec1e0956041a8cfee9e9bae3180e5c0dac). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180264238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50811/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180264237 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180264031 **[Test build #50811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50811/consoleFull)** for PR 10841 at commit [`133a358`](https://github.com/apache/spark/commit/133a358dfe4cb03dc9a4a8f3783dcf32b5508cfc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180644759 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50842/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180644758 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180644692 **[Test build #50842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50842/consoleFull)** for PR 10841 at commit [`91c4f0e`](https://github.com/apache/spark/commit/91c4f0ec1e0956041a8cfee9e9bae3180e5c0dac). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-180232682 **[Test build #50811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50811/consoleFull)** for PR 10841 at commit [`133a358`](https://github.com/apache/spark/commit/133a358dfe4cb03dc9a4a8f3783dcf32b5508cfc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179977274 jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179986856 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179986860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50756/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179144758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50654/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179144751 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179092907 **[Test build #50654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179084732 looks like an internal jenkins issue, jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-179144260 **[Test build #50654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50654/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178879266 **[Test build #50607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50607/consoleFull)** for PR 10841 at commit [`a507f74`](https://github.com/apache/spark/commit/a507f7454c4f5ea872f0f417d9135fee38b4e191). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178881831 **[Test build #50607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50607/consoleFull)** for PR 10841 at commit [`a507f74`](https://github.com/apache/spark/commit/a507f7454c4f5ea872f0f417d9135fee38b4e191). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178881868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178881873 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50607/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178921591 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50619/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178921580 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178923873 looks like an internal jenkins issue, jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178928161 **[Test build #50622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50622/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178962135 **[Test build #50622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50622/consoleFull)** for PR 10841 at commit [`69c5c94`](https://github.com/apache/spark/commit/69c5c9487a3461b08f6eaf3870200b58c9338e32). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178962414 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-178962416 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50622/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-177045628 By the way, there's a section on the design doc attached to https://issues.apache.org/jira/browse/SPARK-10620 that describes the same problem by classifying accumulators into different use cases. I think it's simpler to think about this problem in terms of these use cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10841#issuecomment-177045002
@holdenk Thanks for spending the time to put forth the detailed design. I
agree 100% with the problem and have thought a great deal about how to fix it
myself. However, I personally found the proposed API kind of clunky. It's not
super clear what "consistent" means, especially to someone new to Spark. I
think it would be cleaner to just implement this in the existing classes
themselves and maybe expose only a flag.
It seems that on a high level we want to associate an accumulated value
with a particular computation of an RDD. Can we store something like a map from
RDD ID to a list of accumulated values in each accumulator? Then if the user
calls `acc.value` we just sum all of them up. If the user wants an accumulated
value associated with a particular RDD or even a particular computation then
they can also do that. E.g.
```
class AccumulableInfo[R, T](...) {
private val valuesByRDD = mutable.HashMap[Int, ArrayBuffer[R]]
def value: R = valuesByRDD.values.reduce(mergeThemAll)
def valueByRDD(rddId: Int): R = valuesByRDD(rddId).reduce(mergeThemAll)
}
```
Even this may solve only part of the problem, where the same accumulator
used across different RDDs, but maybe it's a better first step. I may be
missing something but I don't see the need to introduce all these new classes;
the existing set is confusing enough and in the past we've made the mistake of
exposing all of them to the user.
Also, I believe @rxin has some thoughts on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-177047887 Thanks for taking a look @andrewor14 :) This PR (and the JIRA & design doc) are all very much focused on enabling the _data property_ use case for accumulators. I introduced the GenericAccumulable as a parent to Accumulable to try and maintain source compatibility between versions and also limit any cost to users of "regular" accumulators - and since we are coming up on Spark 2.0 that might not be worth the overhead so I'm more than happy to simplify that part away. Or if we change the current accumulator base class in such a way it meets the needs while keeping source compatibility all the better. For the naming of the Consistent Accumulators (or naming of the flag) totally open to ideas - I just didn't have anything else come to mind. So I'm assuming the clunky part of the API your referring to is the "withAccumulator" part of the transformations - which I do feel is pretty clunky myself. My initial draft attempted to avoid using this (I first tried Implementation Option 1 from the design doc ( https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing ) but when multiple transformations are chained together the TaskContext ends up having the RDD id of the inner most RDD. I'd really love to avoid explicitly having the user explicitly pass this information in and if there is a permanent way to do that I'm happy to change this around to give that a shot. I like the idea of the map of RDD id to accumulated value, since an entire RDD might not be computed in the same task though I think we will need to keep track of the value by RDD ID & Partition ID (or at least keep track of value and bitmask of partitions accumulated per RDD ID). cc @squito who did the initial accumulator work and we've gone over the design doc some together. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176324493 **[Test build #50284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50284/consoleFull)** for PR 10841 at commit [`26db2c6`](https://github.com/apache/spark/commit/26db2c6e9dfd21a48803d3820c8d381156b5f01a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176334317 **[Test build #50285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50285/consoleFull)** for PR 10841 at commit [`3e014c7`](https://github.com/apache/spark/commit/3e014c7a07107dd864239b21d67d8fc6511c8529). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176470853 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50295/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176470848 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176300048 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50278/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176300027 **[Test build #50278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50278/consoleFull)** for PR 10841 at commit [`523ba43`](https://github.com/apache/spark/commit/523ba4365a947ccb862bfae158dbcb301ec5163d). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176300039 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176469947 **[Test build #50295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50295/consoleFull)** for PR 10841 at commit [`dabe8cf`](https://github.com/apache/spark/commit/dabe8cff5536dadf46481924ee88154e95a9609e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176473361 I've updated this against master with @andrewor14's Accumulator changes. cc @squito --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176474992 **[Test build #50309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50309/consoleFull)** for PR 10841 at commit [`59c211a`](https://github.com/apache/spark/commit/59c211ab899482946f302aad079bccd5f553baa6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176403742 **[Test build #50285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50285/consoleFull)** for PR 10841 at commit [`3e014c7`](https://github.com/apache/spark/commit/3e014c7a07107dd864239b21d67d8fc6511c8529). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176403957 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50285/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176403954 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176415571 **[Test build #50295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50295/consoleFull)** for PR 10841 at commit [`dabe8cf`](https://github.com/apache/spark/commit/dabe8cff5536dadf46481924ee88154e95a9609e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176429739 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50291/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176429379 **[Test build #50291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50291/consoleFull)** for PR 10841 at commit [`5524bc7`](https://github.com/apache/spark/commit/5524bc74422259b8df1e5e1bda27606764ea34b7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176429734 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176378614 **[Test build #50291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50291/consoleFull)** for PR 10841 at commit [`5524bc7`](https://github.com/apache/spark/commit/5524bc74422259b8df1e5e1bda27606764ea34b7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176506795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50309/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176506787 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176506066 **[Test build #50309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50309/consoleFull)** for PR 10841 at commit [`59c211a`](https://github.com/apache/spark/commit/59c211ab899482946f302aad079bccd5f553baa6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176299436 **[Test build #50278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50278/consoleFull)** for PR 10841 at commit [`523ba43`](https://github.com/apache/spark/commit/523ba4365a947ccb862bfae158dbcb301ec5163d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176325012 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176325002 **[Test build #50284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50284/consoleFull)** for PR 10841 at commit [`26db2c6`](https://github.com/apache/spark/commit/26db2c6e9dfd21a48803d3820c8d381156b5f01a). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-176325014 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50284/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174665371 **[Test build #50010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50010/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174730261 seems unrelated, jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174711126 **[Test build #50010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50010/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174711333 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174711334 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50010/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174734436 **[Test build #50035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50035/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174754737 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174754603 **[Test build #50035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50035/consoleFull)** for PR 10841 at commit [`495c8b6`](https://github.com/apache/spark/commit/495c8b677cef7a38939c09cb7a796c4b1034aea3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174754740 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50035/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-174432893 ping @andrewor14 if you have some time to look at this :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173537102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173537104 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49873/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173536939 **[Test build #49873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49873/consoleFull)** for PR 10841 at commit [`f4100bc`](https://github.com/apache/spark/commit/f4100bc6dd165d025c92fc2853e6b3b075991791). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173506101 **[Test build #49873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49873/consoleFull)** for PR 10841 at commit [`f4100bc`](https://github.com/apache/spark/commit/f4100bc6dd165d025c92fc2853e6b3b075991791). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173525219 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49870/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173525217 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173525109 **[Test build #49870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49870/consoleFull)** for PR 10841 at commit [`cdfd0be`](https://github.com/apache/spark/commit/cdfd0be8ef6d4ee3b4a6656910e1a3cb049e1320). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173500763 **[Test build #49870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49870/consoleFull)** for PR 10841 at commit [`cdfd0be`](https://github.com/apache/spark/commit/cdfd0be8ef6d4ee3b4a6656910e1a3cb049e1320). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173472830 **[Test build #49849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49849/consoleFull)** for PR 10841 at commit [`5ff6b6a`](https://github.com/apache/spark/commit/5ff6b6a6fce974735fd0ad4450fdc382ce8590f9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173425544 That's a good point @codingcat , although the old behavior for regular accumulators is the same I'll add a note about the new API option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173423035 **[Test build #49821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49821/consoleFull)** for PR 10841 at commit [`ce60621`](https://github.com/apache/spark/commit/ce60621dd5048114a1793b5adfad3d72cc423fed). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173423180 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49821/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173423177 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173472900 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49849/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173472899 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173424483 In docs, you need to remove the following sentence > For accumulator updates performed inside actions only, Spark guarantees that each taskâs update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each taskâs update may be applied more than once if tasks or job stages are re-executed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173450266 **[Test build #49849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49849/consoleFull)** for PR 10841 at commit [`5ff6b6a`](https://github.com/apache/spark/commit/5ff6b6a6fce974735fd0ad4450fdc382ce8590f9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173343150 **[Test build #49802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49802/consoleFull)** for PR 10841 at commit [`0915634`](https://github.com/apache/spark/commit/0915634730e7984aa0e086eb10c37e9f68dc0f48). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173345089 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173345095 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49802/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173345083 **[Test build #49802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49802/consoleFull)** for PR 10841 at commit [`0915634`](https://github.com/apache/spark/commit/0915634730e7984aa0e086eb10c37e9f68dc0f48). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173352831 **[Test build #49807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49807/consoleFull)** for PR 10841 at commit [`99acdff`](https://github.com/apache/spark/commit/99acdff674f880e15ecb4e1dbcae935b96bfaeaa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173356288 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49807/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173356239 **[Test build #49807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49807/consoleFull)** for PR 10841 at commit [`99acdff`](https://github.com/apache/spark/commit/99acdff674f880e15ecb4e1dbcae935b96bfaeaa). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173356286 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173396095 **[Test build #49821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49821/consoleFull)** for PR 10841 at commit [`ce60621`](https://github.com/apache/spark/commit/ce60621dd5048114a1793b5adfad3d72cc423fed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/10841
[SPARK-12469][CORE][RFC/WIP] Add Consistent Accumulators for Spark
This is an initial PR illustrating one of the possible approaches for
providing consistent accumulators in Spark.
Tasks executed on Spark workers are unable to modify values from the
driver, and accumulators are the one exception for this. Accumulators in Spark
are implemented in such a way that when a stage is recomputed (say for cache
eviction) the accumulator will be updated a second time. This makes
accumulators inside of transformations more difficult to use for things like
counting invalid records (one of the primary potential use cases of collecting
side information during a transformation). However in some cases this counting
during re-evaluation is exactly the behaviour we want (say in tracking total
execution time for a particular function). Spark would benefit from a version
of accumulators which did not double count even if stages were re-executed.
Motivating example:
```
val parseTime = sc.accumulator(0L)
val parseFailures = sc.accumulator(0L)
val parsedData = sc.textFile(...).flatMap { line =>
val start = System.currentTimeMillis()
val parsed = Try(parse(line))
if (parsed.isFailure) parseFailures += 1
parseTime += System.currentTimeMillis() - start
parsed.toOption
}
parsedData.cache()
val resultA = parsedData.map(...).filter(...).count()
// some intervening code. Almost anything could happen here -- some of
parsedData may
// get kicked out of the cache, or an executor where data was cached might
get lost
val resultB = parsedData.filter(...).map(...).flatMap(...).count()
// now we look at the accumulators
```
Here we would want parseFailures to only have been added to once for every
line which failed to parse. Unfortunately, the current Spark accumulator API
doesnât support the current parseFailures use case since if some data had
been evicted its possible that it will be double counted.
See the full design document at
https://docs.google.com/document/d/1lR_l1g3zMVctZXrcVjFusq2iQVpr4XvRK_UUDsDr6nk/edit?usp=sharing
cc @squito
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-12469-consistent-accumulators-for-spark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10841.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10841
commit 7c9f874d94085f60089cda8ab3b6c451bf793a1a
Author: Holden Karau
Date: 2015-12-28T19:59:30Z
Start adding the proposed external API for consistent accumulators
commit fce92981371ddd7af8ad28f30537fb473bbee378
Author: Holden Karau
Date: 2015-12-28T19:59:47Z
Start keeping track of the required information in the taskcontext
commit c0e2bfd9d3d5f22e9ea217a500b67e84679f519d
Author: Holden Karau
Date: 2015-12-30T00:29:53Z
A bit more progress towards consistent accumulators, add an add function
which checks the bitset, throw an exception on merges (perhaps this isn't
reasonable and then we need to keep all of the previous values - but a quick
skim looks like its intended to be a user accesiable function so just don't
support).
commit e9c287f682cbefc531a8fc84b00a472f860c9d06
Author: Holden Karau
Date: 2015-12-30T01:51:53Z
Waaait that was not thinking clearly. As we go with a consistent
accumulator adding values add them to a hashmap of pending values to be merged
based on the rdd id & partition id - then driver side when merging the
accumulators check each rdd & partition id individually against the rdd &
bitset of processed partitions for that rdd adding to a single accumulator.
(This way we don't need to keep all of the values around forever but if we say
have first() and then foreach() we can add the value for partition one then
when the second task happens and we can skip just the adding partion one. Up
next: start adding some simple tests for this to see if it works
commit cf32dc5d543d4c0bb883e00692d21f22a7470737
Author: Holden Karau
Date: 2016-01-06T07:45:18Z
Merge branch 'master' into SPARK-12469-consistent-accumulators-for-spark
commit d9354b12977bdaa2bcd82730fb34bdae490ab02d
Author: Holden Karau
Date: 2016-01-06T20:26:13Z
add consistentValue (temporary) to get the consistent value and fix the
accumulator to work even when no merge happens
commit 28a5754b654441ad8904f6a8f6e4a06db3908541
Author: Holden Karau
Date: 2016-01-06T20:58:07Z
Introduce a GenericAccumulable to allow for Accumulables which
[GitHub] spark pull request: [SPARK-12469][CORE][RFC/WIP] Add Consistent Ac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10841#issuecomment-173026556 **[Test build #49723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49723/consoleFull)** for PR 10841 at commit [`3f96429`](https://github.com/apache/spark/commit/3f96429b5c6a6c5596d70a9d15de74ae6203c970). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
