Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22010
If this is not yet in 2.4 it shouldnât be merged now.
On Wed, Oct 10, 2018 at 10:57 AM Holden Karau
wrote:
> Open question: is this suitable for branch-2.4 since it predates the
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Open question: is this suitable for branch-2.4 since it predates the branch
cut or not? (I know we've gone back and forth on how we do that).
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
thanks, merging to master!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96680/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96680 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96680/testReport)**
for PR 22010 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22010
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96680 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96680/testReport)**
for PR 22010 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96669 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96669/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96669/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96669 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96669/testReport)**
for PR 22010 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96652/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96652 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96652/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96652 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96652/testReport)**
for PR 22010 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96641/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96641 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96641/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96641 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96641/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
I'll leave this until Friday morning (pacific) in case anyone has last
minute comments. cc @rxin / @HyukjinKwon / @mgaido91
---
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
So by running `sc.parallelize(1.to(1000)).map(x => (x % 10,
x)).sortByKey().distinct().count()` in 2.3.0 and my PR we can see the
difference:
**
for PR 22010 at commit
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Did another quick micro benchmark on a small cluster:
```scala
import org.apache.spark.util.collection.ExternalAppendOnlyMap
def removeDuplicatesInPartition(partition:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #96583 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96583/testReport)**
for PR 22010 at commit
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
@cloud-fan yeah that's totally an option. Since @rxin asked for it to use
`reduceByKey` I went with that approach, but I'd be happy to use the
`ExternalAppendOnlyMap` if that's ok with folks.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
I think this works, can we post some Spark web UI screenshots to confirm
the shuffle is indeed eliminated?
BTW one idea to simplify the implementation:
```
def
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
@rxin So that RDD could not exist with a known partitioner (regardless of
range-based or hash based the partitioner must be deterministic so two elements
with the same key must go to the same
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22010
Actually @holdenk is this change even correct? RDD.distinct is not key
based. It is based on the value of the elements in RDD. Even if `numPartitions
== partitions.length`, it doesn't mean the RDD is
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Hey @rxin & @cloud-fan I'd really appreciate your input on the tricks I did
to keep the partioniner information present -- is this the right approach?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95768/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #95768 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95768/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95767/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #95767 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95767/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #95768 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95768/testReport)**
for PR 22010 at commit
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Updated to use reduceByKey. I'd really appreciate feedback on if adding the
param to `MapPartitionsRDD` was the way to go or if I should sub-class it
instead.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #95767 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95767/testReport)**
for PR 22010 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22010
thanks for checking @rxin @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22010
I am sorry guys. I rushed to take a look.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22010
While this saves a shuffle, but the algorithm becomes different. Previously
we use the shuffe aggregator, which stores data in a `ExternalAppendOnlyMap`.
Now we use a scala set, which may OOM.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/22010
Thanks for pinging. Please don't merge this until you've addressed the OOM
issue. The aggregators were created to handle incoming data larger than size of
memory. We should never use a Scala or Java
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
I think this is pretty clearly a win, but since it's been awhile since I
did anything in core I'll leave this until Friday morning (pacific) in-case any
of the committers who've been working there
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
I did a quick micro-benchmark on this and got:
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
>
> import scala.collection.{mutable, Map}
> def
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
@HyukjinKwon sure, I'll do a micro benchmark sometime this week.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22010
Logically looks right but would you mind if I ask a simple benchmark
@holdenk just to make everything clear?
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94634/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94634 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94634/testReport)**
for PR 22010 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94634 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94634/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/22010
Test failure is streaming timeout, likely unrelated. Jenkins retest this
please.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94607/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94607 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94607/testReport)**
for PR 22010 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94607 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94607/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22010
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94577/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94577 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94577/testReport)**
for PR 22010 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94577 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94577/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94303/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94303 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94303/testReport)**
for PR 22010 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22010
**[Test build #94303 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94303/testReport)**
for PR 22010 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22010
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
92 matches
Mail list logo
