[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9066 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147268583 @yhuai thanks for reviewing. I've addressed your comment by removing the unnecessary `copy()` calls and the tests are still passing, so I'm going to merge this now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147258602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43548/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147258600 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147258336 [Test build #43548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43548/console) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147257359 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43551/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147257358 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147257322 [Test build #43551 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43551/console) for PR 9066 at commit [`ef53862`](https://github.com/apache/spark/commit/ef538621f7415a8502615181f6fb08721a809ff6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147249986 [Test build #43548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43548/consoleFull) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147249549 [Test build #43551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43551/consoleFull) for PR 9066 at commit [`ef53862`](https://github.com/apache/spark/commit/ef538621f7415a8502615181f6fb08721a809ff6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147249269 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147249267 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147246295 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9066#discussion_r41714859
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala
---
@@ -101,18 +117,16 @@ class SortBasedAggregationIterator(
// The search will stop when we see the next group or there is no
// input row left in the iter.
-var hasNext = inputKVIterator.next()
-while (!findNextPartition && hasNext) {
+while (!findNextPartition && inputIterator.hasNext) {
// Get the grouping key.
- val groupingKey = inputKVIterator.getKey
- val currentRow = inputKVIterator.getValue
+ val inputRow = inputIterator.next()
+ val groupingKey = groupingKeyProjection(inputRow).copy()
+ val currentRow = inputRow.copy()
--- End diff --
Good catch. We only need these copies in `initialize` since we're buffering
the result there. I'll fix this shortly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147241117 Looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9066#discussion_r41714812
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala
---
@@ -101,18 +117,16 @@ class SortBasedAggregationIterator(
// The search will stop when we see the next group or there is no
// input row left in the iter.
-var hasNext = inputKVIterator.next()
-while (!findNextPartition && hasNext) {
+while (!findNextPartition && inputIterator.hasNext) {
// Get the grouping key.
- val groupingKey = inputKVIterator.getKey
- val currentRow = inputKVIterator.getValue
+ val inputRow = inputIterator.next()
+ val groupingKey = groupingKeyProjection(inputRow).copy()
+ val currentRow = inputRow.copy()
--- End diff --
Maybe we can avoid of these two `copy`s?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147239086 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147239037 jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147171456 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147171458 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43538/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147171452 [Test build #43538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43538/console) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147167795 [Test build #43538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43538/consoleFull) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147167003 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147161394 Looks ok to me. I will leave it to @yhuai for the detailed review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147161343 /cc @yhuai @rxin for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147158226 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147158207 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147156067 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147156068 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43535/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147156060 [Test build #43535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43535/console) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147155599 [Test build #43535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43535/consoleFull) for PR 9066 at commit [`2b8bf94`](https://github.com/apache/spark/commit/2b8bf947b0434146131b3af4a314cc89ace8a874). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147153492 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9066#issuecomment-147153450 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11053] Remove use of KVIterator in Sort...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/9066 [SPARK-11053] Remove use of KVIterator in SortBasedAggregationIterator SortBasedAggregationIterator uses a KVIterator interface in order to process input rows as key-value pairs, but this use of KVIterator is unnecessary, slightly complicates the code, and might hurt performance. This patch refactors this code to remove the use of this extra layer of iterator wrapping and simplifies other parts of the code in the process. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark sort-iterator-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9066.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9066 commit 91c88ebaf888cf4ca1885af5fa18d2f1c8a4f926 Author: Josh Rosen Date: 2015-10-11T03:19:37Z Simplify wrapping of iterator into KVIterator. commit 2b8bf947b0434146131b3af4a314cc89ace8a874 Author: Josh Rosen Date: 2015-10-11T04:17:29Z Remove KVIterator from SortBasedAggregationIterator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
