Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11327
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216658381
Merging this into master, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does no
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216647234
Test build #2965 has finished successfully so I'm going by that. The other
one had another unit test failure that is unrelated.
---
If your project is set up for it
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216646855
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216646851
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216646622
**[Test build #57653 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57653/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216644889
**[Test build #2965 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2965/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216621319
**[Test build #57653 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57653/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://gi
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216620528
test failure is in ExternalAppendOnlyMapSuite which is unrelated. I'll kick
jenkins again.
---
If your project is set up for it, you can reply to this email and have
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216620552
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216613204
**[Test build #2965 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2965/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216612214
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216612218
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216611966
**[Test build #57644 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57644/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://g
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216603791
LGTM, pending tests. It's great to have 200X speedup, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as w
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216585173
**[Test build #57644 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57644/consoleFull)**
for PR 11327 at commit
[`305a7db`](https://gi
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216584419
thanks for the review, made changes and updated description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216358709
@tgravescs Could you also update the description of reflect the new changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61801165
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -334,8 +332,41 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Doubl
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61801132
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -334,8 +332,41 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Doubl
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61798932
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -320,7 +317,8 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Double
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61798850
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -289,10 +284,12 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Doub
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216353266
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216353258
**[Test build #57556 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57556/consoleFull)**
for PR 11327 at commit
[`f012cd5`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216353264
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216352965
**[Test build #57556 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57556/consoleFull)**
for PR 11327 at commit
[`f012cd5`](https://gi
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61798109
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -169,43 +169,41 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Doub
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r61797836
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -169,43 +169,41 @@ private class DefaultPartitionCoalescer(val
balanceSlack: Doub
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216350597
@tgravescs That's great, could you fix the style?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216339716
**[Test build #57550 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57550/consoleFull)**
for PR 11327 at commit
[`2eff583`](https://g
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216339722
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216339720
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216339359
Ok, I replaced the location iterator and now get all the preferred
locations up front. This made the run time of the this go from around a minute
down to around 6 se
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-216339363
**[Test build #57550 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57550/consoleFull)**
for PR 11327 at commit
[`2eff583`](https://gi
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-200963748
I think the current implementation does not handle location changing, and
we can't.
---
If your project is set up for it, you can reply to this email and have your
repl
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-200961327
thanks for the feedback. I'm fine with that change and actually had
considered it, I just wasn't sure if the intention of the location iterator was
to handle the loc
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-200959288
@tgravescs The current change is good for your case, I'm thinking that
maybe we could do better.
The LocationIterator has a bad smell, it may call getPreferredLo
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r57363138
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -324,6 +319,40 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: R
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r57361560
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -324,6 +319,40 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r57361348
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -324,6 +319,40 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r57361377
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -324,6 +319,40 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-200929309
@tgravescs I'm reviewing this now, sorry for the late.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-200904458
ping @davies @rxin Any chance I can get review on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as we
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-193340890
@tgravescs I did not have enough time to look into the details yet (not
familar this part), sorry for the delay.
---
If your project is set up for it, you can reply to
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-193293645
ping @davies was there any other concern or does this look good?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54446793
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[_],
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-190241612
Are there any other comments about functionality?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54418281
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RD
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54375032
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[_],
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54115176
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RD
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54116119
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[_
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11327#discussion_r54019069
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
@@ -192,7 +192,8 @@ private class PartitionCoalescer(maxPartitions: Int,
prev: RDD[_
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-188142561
cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabl
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187969886
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187969889
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187969631
**[Test build #51799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51799/consoleFull)**
for PR 11327 at commit
[`8665114`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187921218
**[Test build #51799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51799/consoleFull)**
for PR 11327 at commit
[`8665114`](https://gi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187891776
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187891779
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187891764
**[Test build #51789 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51789/consoleFull)**
for PR 11327 at commit
[`c9eb032`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187891331
**[Test build #51789 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51789/consoleFull)**
for PR 11327 at commit
[`c9eb032`](https://gi
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187885309
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187883014
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187883016
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187875082
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187875090
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187875072
**[Test build #51786 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51786/consoleFull)**
for PR 11327 at commit
[`afe14dc`](https://g
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187874407
**[Test build #51786 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51786/consoleFull)**
for PR 11327 at commit
[`afe14dc`](https://gi
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187858375
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187851796
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your projec
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11327#issuecomment-187851798
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/
71 matches
Mail list logo
