[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-3562:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Navis for your persistence on this one!
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Fix For: 0.12.0
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch,
HIVE-3562.D5967.6.patch, HIVE-3562.D5967.7.patch, HIVE-3562.D5967.8.patch,
HIVE-3562.D5967.9.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.7.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Addressed some comments
Reviewers: ashutoshc, JIRA, tarball
REVISION DETAIL
https://reviews.facebook.net/D5967
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D5967?vs=38379id=39009#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/build.xml
ql/ivy.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/queries/clientpositive/limit_pushdown_negative.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
ql/src/test/results/clientpositive/limit_pushdown_negative.q.out
To: JIRA, tarball, ashutoshc, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch,
HIVE-3562.D5967.6.patch, HIVE-3562.D5967.7.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.8.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Missed ASF header
Removed unnecessary array creation in RS
Reviewers: ashutoshc, JIRA, tarball
REVISION DETAIL
https://reviews.facebook.net/D5967
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D5967?vs=39009id=39015#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/build.xml
ql/ivy.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/queries/clientpositive/limit_pushdown_negative.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
ql/src/test/results/clientpositive/limit_pushdown_negative.q.out
To: JIRA, tarball, ashutoshc, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch,
HIVE-3562.D5967.6.patch, HIVE-3562.D5967.7.patch, HIVE-3562.D5967.8.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.9.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Addressed comments
Reviewers: ashutoshc, JIRA, tarball
REVISION DETAIL
https://reviews.facebook.net/D5967
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D5967?vs=39015id=39051#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/build.xml
ql/ivy.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/queries/clientpositive/limit_pushdown_negative.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
ql/src/test/results/clientpositive/limit_pushdown_negative.q.out
To: JIRA, tarball, ashutoshc, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch,
HIVE-3562.D5967.6.patch, HIVE-3562.D5967.7.patch, HIVE-3562.D5967.8.patch,
HIVE-3562.D5967.9.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.6.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Rebased to trunk
Reviewers: tarball, JIRA
REVISION DETAIL
https://reviews.facebook.net/D5967
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D5967?vs=32943id=38379#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/build.xml
ql/ivy.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
To: JIRA, tarball, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch,
HIVE-3562.D5967.6.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3562:
Status: Patch Available (was: Open)
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch, HIVE-3562.D5967.5.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3562:
-
Status: Open (was: Patch Available)
can you refresh ?
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3562:
Status: Patch Available (was: Open)
Passed all tests
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.4.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
1. Used Heap for ORDER BY and Map for GROUP BY
2. Added tests for spill/break
3. Changed to use percentage for memory threshold
Reviewers: tarball, JIRA
REVISION DETAIL
https://reviews.facebook.net/D5967
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D5967?vs=24861id=30483#toc
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/build.xml
ql/ivy.xml
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHashForGBY.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
To: JIRA, tarball, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3562:
-
Status: Open (was: Patch Available)
comments
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3562:
Status: Patch Available (was: Open)
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.3.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Reviewers: JIRA, tarball
Addressed comments
Prevent multi-GBY single-RS case
REVISION DETAIL
https://reviews.facebook.net/D5967
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
To: JIRA, tarball, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch,
HIVE-3562.D5967.3.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3562:
-
Status: Open (was: Patch Available)
comments on phabricator
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3562:
Status: Patch Available (was: Open)
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.2.patch
navis updated the revision HIVE-3562 [jira] Some limit can be pushed down to
map stage.
Reviewers: JIRA, tarball
Addressed some comments (submitting for fail-fast review)
REVISION DETAIL
https://reviews.facebook.net/D5967
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
To: JIRA, tarball, navis
Cc: njain
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Namit Jain updated HIVE-3562:
-
Status: Open (was: Patch Available)
comments on phabricator
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-3562:
--
Attachment: HIVE-3562.D5967.1.patch
navis requested code review of HIVE-3562 [jira] Some limit can be pushed down
to map stage.
Reviewers: JIRA
DPAL-1910 Some limit can be pushed down to map stage
Queries with limit clause (with reasonable number), for example
select * from src order by key limit 10;
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D5967
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
ql/src/test/queries/clientpositive/limit_pushdown.q
ql/src/test/results/clientpositive/limit_pushdown.q.out
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/14199/
To: JIRA, navis
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3562:
Status: Patch Available (was: Open)
Some limit can be pushed down to map stage
--
Key: HIVE-3562
URL: https://issues.apache.org/jira/browse/HIVE-3562
Project: Hive
Issue Type: Bug
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-3562.D5967.1.patch
Queries with limit clause (with reasonable number), for example
{noformat}
select * from src order by key limit 10;
{noformat}
makes operator tree,
TS-SEL-RS-EXT-LIMIT-FS
But LIMIT can be partially calculated in RS, reducing size of shuffling.
TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
