[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[
https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085961#comment-14085961
]
Hive QA commented on HIVE-7527:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12659822/HIVE-7527.2-spark.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5826 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/12/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/12/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-12/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12659822
> Support order by and sort by on Spark
> -
>
> Key: HIVE-7527
> URL: https://issues.apache.org/jira/browse/HIVE-7527
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Rui Li
> Fix For: spark-branch
>
> Attachments: HIVE-7527-spark.patch, HIVE-7527.2-spark.patch
>
>
> Currently Hive depends completely on MapReduce's sorting as part of shuffling
> to achieve order by (global sort, one reducer) and sort by (local sort).
> Spark has a sort by transformation in different variations that can used to
> support Hive's order by and sort by. However, we still need to evaluate
> weather Spark's sortBy can achieve the same functionality inherited from
> MapReduce's shuffle sort.
> Currently Hive on Spark should be able to run simple sort by or order by, by
> changing the currently partitionBy to sortby. This is the way to verify
> theories. Complete solution will not be available until we have complete
> SparkPlanGenerator.
> There is also a question of how we determine that there is order by or sort
> by by just looking at the operator tree, from which Spark task is created.
> This is the responsibility of SparkPlanGenerator, but we need to have an idea.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085836#comment-14085836 ] Rui Li commented on HIVE-7527: -- Thanks [~brocknoland] for the review :-) > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Fix For: spark-branch > > Attachments: HIVE-7527-spark.patch, HIVE-7527.2-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085831#comment-14085831 ] Brock Noland commented on HIVE-7527: Actually since we don't yet have any tests that can be impacted I will verify this doesn't break the build and then commit. > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch, HIVE-7527.2-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085823#comment-14085823 ] Brock Noland commented on HIVE-7527: +1 I'll let the tests run and then commit this. > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch, HIVE-7527.2-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085817#comment-14085817 ] Rui Li commented on HIVE-7527: -- Hi [~brocknoland] I've updated the patch. Please help to review, thanks. > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch, HIVE-7527.2-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085691#comment-14085691 ] Xuefu Zhang commented on HIVE-7527: --- [~lirui] It should be okay, if the conflicts are just about code (non-functional). Whichever gets committed first, the latter one would have to rebase. > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085680#comment-14085680 ] Rui Li commented on HIVE-7527: -- I also noticed that the numPartitions stuff has some conflicts with HIVE-7567. We may want to decide how to do it. cc [~xuefuz] [~chengxiang li] > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085675#comment-14085675 ] Rui Li commented on HIVE-7527: -- Thanks [~brocknoland], I've created the request at: https://reviews.apache.org/r/24290/ > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085654#comment-14085654 ] Brock Noland commented on HIVE-7527: Hi, You can leave that empty and then post the RB link here. Otherwise you can specify: Group: Hive People: Brock Brock > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085642#comment-14085642 ] Rui Li commented on HIVE-7527: -- Hi [~brocknoland], how to specify reviewer/group to publish a request on RB? > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085002#comment-14085002 ] Brock Noland commented on HIVE-7527: Nice work! Can you add a RB item? http://reviews.apache.org > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang >Assignee: Rui Li > Attachments: HIVE-7527-spark.patch > > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[
https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084763#comment-14084763
]
Hive QA commented on HIVE-7527:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12659646/HIVE-7527-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5826 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/10/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/10/console
Test logs:
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-10/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12659646
> Support order by and sort by on Spark
> -
>
> Key: HIVE-7527
> URL: https://issues.apache.org/jira/browse/HIVE-7527
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Rui Li
> Attachments: HIVE-7527-spark.patch
>
>
> Currently Hive depends completely on MapReduce's sorting as part of shuffling
> to achieve order by (global sort, one reducer) and sort by (local sort).
> Spark has a sort by transformation in different variations that can used to
> support Hive's order by and sort by. However, we still need to evaluate
> weather Spark's sortBy can achieve the same functionality inherited from
> MapReduce's shuffle sort.
> Currently Hive on Spark should be able to run simple sort by or order by, by
> changing the currently partitionBy to sortby. This is the way to verify
> theories. Complete solution will not be available until we have complete
> SparkPlanGenerator.
> There is also a question of how we determine that there is order by or sort
> by by just looking at the operator tree, from which Spark task is created.
> This is the responsibility of SparkPlanGenerator, but we need to have an idea.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7527) Support order by and sort by on Spark
[ https://issues.apache.org/jira/browse/HIVE-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14076187#comment-14076187 ] Rui Li commented on HIVE-7527: -- Hi [~xuefuz], I tried to run order by queries using spark's sortByKey transformation but it seems the result is incorrect. I inserted the sortByKey between HiveMapFunction and HiveReduceFunction (in substitute of partitionBy). Wondering if this is the right way to do it... I detect the order by by looking at the parent ReduceSink when a ReduceWork is created and connected to a MapWork. It worked for my simple cases :) > Support order by and sort by on Spark > - > > Key: HIVE-7527 > URL: https://issues.apache.org/jira/browse/HIVE-7527 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Xuefu Zhang > > Currently Hive depends completely on MapReduce's sorting as part of shuffling > to achieve order by (global sort, one reducer) and sort by (local sort). > Spark has a sort by transformation in different variations that can used to > support Hive's order by and sort by. However, we still need to evaluate > weather Spark's sortBy can achieve the same functionality inherited from > MapReduce's shuffle sort. > Currently Hive on Spark should be able to run simple sort by or order by, by > changing the currently partitionBy to sortby. This is the way to verify > theories. Complete solution will not be available until we have complete > SparkPlanGenerator. > There is also a question of how we determine that there is order by or sort > by by just looking at the operator tree, from which Spark task is created. > This is the responsibility of SparkPlanGenerator, but we need to have an idea. -- This message was sent by Atlassian JIRA (v6.2#6252)
