Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r61227913
--- Diff:
core/src/test/scala/org/apache/spark/executor/TaskMetricsSuite.scala ---
@@ -214,46 +214,14 @@ class TaskMetricsSuite extends SparkFunSuite {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-215024478
**[Test build #57111 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57111/consoleFull)**
for PR 12612 at commit
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r61227652
--- Diff: core/src/test/scala/org/apache/spark/AccumulatorSuite.scala ---
@@ -257,38 +249,33 @@ class AccumulatorSuite extends SparkFunSuite with
Matchers
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-215024490
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-215024488
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-215023279
**[Test build #57111 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57111/consoleFull)**
for PR 12612 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r61217219
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r61103191
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214754273
**[Test build #57000 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57000/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214754278
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214754282
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214753658
**[Test build #57000 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57000/consoleFull)**
for PR 12612 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214653257
**[Test build #56981 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56981/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214653264
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214653262
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214652836
**[Test build #56981 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56981/consoleFull)**
for PR 12612 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r61014595
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60995516
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60994954
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60994309
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60942830
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214427973
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214427960
**[Test build #56900 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56900/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214427979
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214425748
**[Test build #56900 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56900/consoleFull)**
for PR 12612 at commit
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60861038
--- Diff: core/src/main/scala/org/apache/spark/Accumulator.scala ---
@@ -17,121 +17,281 @@
package org.apache.spark
+import java.{lang =>
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60860997
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -287,11 +258,10 @@ private[spark] object TaskMetrics extends Logging {
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60860782
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -287,11 +258,10 @@ private[spark] object TaskMetrics extends Logging {
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60859262
--- Diff: core/src/main/scala/org/apache/spark/Accumulator.scala ---
@@ -17,121 +17,281 @@
package org.apache.spark
+import java.{lang
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60853135
--- Diff: core/src/main/scala/org/apache/spark/Accumulator.scala ---
@@ -17,121 +17,281 @@
package org.apache.spark
+import java.{lang =>
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60853137
--- Diff: core/src/main/scala/org/apache/spark/Accumulator.scala ---
@@ -17,121 +17,281 @@
package org.apache.spark
+import java.{lang =>
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60852232
--- Diff: core/src/main/scala/org/apache/spark/Accumulator.scala ---
@@ -17,121 +17,281 @@
package org.apache.spark
+import java.{lang
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60852131
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -287,11 +258,10 @@ private[spark] object TaskMetrics extends Logging {
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214038273
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214038271
**[Test build #56863 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56863/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214038272
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214038184
**[Test build #56863 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56863/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214036804
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214036788
**[Test build #56862 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56862/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214036800
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214036538
**[Test build #56862 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56862/consoleFull)**
for PR 12612 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214032265
**[Test build #56861 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56861/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214032267
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214032266
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214032198
**[Test build #56861 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56861/consoleFull)**
for PR 12612 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214031701
**[Test build #56860 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56860/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214031706
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214031704
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214031631
**[Test build #56860 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56860/consoleFull)**
for PR 12612 at commit
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214017619
cc @rxin , several questions need to be discussed:
1. should we provide the metadata while creating accumulators instead of
registration? e.g., `id` can be
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001917
**[Test build #56850 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56850/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001929
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001926
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001696
**[Test build #56850 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56850/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001178
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001174
**[Test build #56848 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56848/consoleFull)**
for PR 12612 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001179
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-214001078
**[Test build #56848 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56848/consoleFull)**
for PR 12612 at commit
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60822740
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213465626
The `SQLMetrics` is a good example to show how simple the new API is
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12612#discussion_r60753108
--- Diff: core/src/main/scala/org/apache/spark/NewAccumulator.scala ---
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213463937
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213463939
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213463929
**[Test build #56704 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56704/consoleFull)**
for PR 12612 at commit
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213463689
cc @rxin, I'm wondering if `AccumulatorMetadata` is necessary, it makes
sense to provide the id during registration, but seems it's more convenient to
provide the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12612#issuecomment-213463543
**[Test build #56704 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56704/consoleFull)**
for PR 12612 at commit
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/12612
[SPARK-14654][CORE][WIP] New accumulator API
## What changes were proposed in this pull request?
This PR introduces a new accumulator API which is much simpler than before:
1.
67 matches
Mail list logo
