[ https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533638#comment-13533638 ]
Phabricator commented on HIVE-3562: ----------------------------------- njain has commented on the revision "HIVE-3562 [jira] Some limit can be pushed down to map stage". The general direction looks OK INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:79 TODO ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:45 spelling: operator ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:79 Followup: add a new method in Operator. ql/src/test/queries/clientpositive/limit_pushdown.q:26 Looks like this optimization should also help if the limit is in a sub-query: Can you add a test ? something like: select .. from (select key, count(1) from src group by key order by key limit 2) subq join (select key, count(1) from src group by key order by key limit 2) subq2 .. The optimization should be applied to both the sub-queries REVISION DETAIL https://reviews.facebook.net/D5967 BRANCH DPAL-1910 To: JIRA, tarball, navis Cc: njain > Some limit can be pushed down to map stage > ------------------------------------------ > > Key: HIVE-3562 > URL: https://issues.apache.org/jira/browse/HIVE-3562 > Project: Hive > Issue Type: Bug > Reporter: Navis > Assignee: Navis > Priority: Trivial > Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch > > > Queries with limit clause (with reasonable number), for example > {noformat} > select * from src order by key limit 10; > {noformat} > makes operator tree, > TS-SEL-RS-EXT-LIMIT-FS > But LIMIT can be partially calculated in RS, reducing size of shuffling. > TS-SEL-RS(TOP-N)-EXT-LIMIT-FS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira