[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165153#comment-13165153 ] Hudson commented on HIVE-1003: -- Integrated in Hive-trunk-h0.21 #1131 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1131/]) HIVE-1003 [jira] optimize metadata only queries (Namit Jain via Yongqiang He) Summary: testing Queries like: select max(ds) from T where ds is a partitioning column should be optimized. Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: njain, heyongqiang Differential Revision: 105 heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211767 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java * /hive/trunk/ql/src/test/queries/clientpositive/partition_schema1.q * /hive/trunk/ql/src/test/results/clientpositive/alter_table_serde.q.out * /hive/trunk/ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_schema1.q.out > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.8.0, 0.9.0 > > Attachments: D105.1.patch, D105.2.patch, HIVE-1003.1.patch, > HIVE-1003.D105.3.patch, hive.1003.2.patch, hive.1003.3.patch, > hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165068#comment-13165068 ] Hudson commented on HIVE-1003: -- Integrated in Hive-trunk-h0.23.0 #7 (See [https://builds.apache.org/job/Hive-trunk-h0.23.0/7/]) HIVE-1003 [jira] optimize metadata only queries (Namit Jain via Yongqiang He) Summary: testing Queries like: select max(ds) from T where ds is a partitioning column should be optimized. Test Plan: EMPTY Reviewers: JIRA, heyongqiang Reviewed By: heyongqiang CC: njain, heyongqiang Differential Revision: 105 heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211767 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java * /hive/trunk/ql/src/test/queries/clientpositive/partition_schema1.q * /hive/trunk/ql/src/test/results/clientpositive/alter_table_serde.q.out * /hive/trunk/ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out * /hive/trunk/ql/src/test/results/clientpositive/partition_schema1.q.out > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.8.0, 0.9.0 > > Attachments: D105.1.patch, D105.2.patch, HIVE-1003.1.patch, > HIVE-1003.D105.3.patch, hive.1003.2.patch, hive.1003.3.patch, > hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165037#comment-13165037 ] Phabricator commented on HIVE-1003: --- njain has committed the revision "HIVE-1003 [jira] optimize metadata only queries". REVISION DETAIL https://reviews.facebook.net/D105 COMMIT https://reviews.facebook.net/rHIVE1211767 > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.8.0, 0.9.0 > > Attachments: D105.1.patch, D105.2.patch, HIVE-1003.1.patch, > HIVE-1003.D105.3.patch, hive.1003.2.patch, hive.1003.3.patch, > hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165033#comment-13165033 ] Phabricator commented on HIVE-1003: --- heyongqiang has accepted the revision "HIVE-1003 [jira] optimize metadata only queries". tests passed REVISION DETAIL https://reviews.facebook.net/D105 > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.8.0, 0.9.0 > > Attachments: D105.1.patch, D105.2.patch, HIVE-1003.1.patch, > HIVE-1003.D105.3.patch, hive.1003.2.patch, hive.1003.3.patch, > hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140649#comment-13140649 ] Hudson commented on HIVE-1003: -- Integrated in Hive-trunk-h0.21 #1048 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1048/]) HIVE-1003: optimize metadata only queries (Marcin Kurczych, Namit Jain via He Yongqiang) heyongqiang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195577 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/OneNullRowInputFormat.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFType.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/test/queries/clientpositive/metadataonly1.q * /hive/trunk/ql/src/test/results/clientpositive/metadataonly1.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/NullStructSerDe.java > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Fix For: 0.9.0 > > Attachments: D105.1.patch, HIVE-1003.1.patch, hive.1003.2.patch, > hive.1003.3.patch, hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140349#comment-13140349 ] He Yongqiang commented on HIVE-1003: committed, thanks Marcin and Namit! > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: D105.1.patch, HIVE-1003.1.patch, hive.1003.2.patch, > hive.1003.3.patch, hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138818#comment-13138818 ] He Yongqiang commented on HIVE-1003: looks good to me, running tests > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: D105.1.patch, HIVE-1003.1.patch, hive.1003.2.patch, > hive.1003.3.patch, hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138515#comment-13138515 ] Namit Jain commented on HIVE-1003: -- addressed comments > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: D105.1.patch, HIVE-1003.1.patch, hive.1003.2.patch, > hive.1003.3.patch, hive.1003.4.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138035#comment-13138035 ] Phabricator commented on HIVE-1003: --- njain has commented on the revision "HIVE-1003 [jira] optimize metadata only queries". testing integration REVISION DETAIL https://reviews.facebook.net/D105 > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: D105.1.patch, HIVE-1003.1.patch, hive.1003.2.patch, > hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134458#comment-13134458 ] Namit Jain commented on HIVE-1003: -- @Ashutosh, moving to physical optimizer should also fix the query in HIVE-2119 > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch, hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134344#comment-13134344 ] He Yongqiang commented on HIVE-1003: 1) move the optimizer to physical optimizer. 2) after 1), changes in GenMapRedUtils also need to be moved. 3) add a testcase which uses CombineHiveInputFormat. 4) also test virtual_column.q nitpicks: 1) getCategory() in NullStructSerDe's getObjectInspector should return Struct 2) remove import of "HiveNullValueSequenceFileOutputFormat" and "OneNullRowInputFormat" from SemanticAnalyzer.java > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch, hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133829#comment-13133829 ] Namit Jain commented on HIVE-1003: -- @Ashutosh, that query is not optimized - Let us keep HIVE-2119 open and work on that as a followup. > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch, hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133818#comment-13133818 ] Namit Jain commented on HIVE-1003: -- @Ashutosh, it should - i haven't tried it yet. Let me confirm tomorrow > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch, hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133759#comment-13133759 ] Ashutosh Chauhan commented on HIVE-1003: @Namit, Will second query on HIVE-2119 will also get optimized with this? > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch, hive.1003.3.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132843#comment-13132843 ] He Yongqiang commented on HIVE-1003: will take a look. > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch, hive.1003.2.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1003) optimize metadata only queries
[ https://issues.apache.org/jira/browse/HIVE-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130785#comment-13130785 ] Marcin Kurczych commented on HIVE-1003: --- I have finished my internship at Facebook. If you want to contact me you can reach me at Marcin Kurczych . Thanks! > optimize metadata only queries > -- > > Key: HIVE-1003 > URL: https://issues.apache.org/jira/browse/HIVE-1003 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Marcin Kurczych > Attachments: HIVE-1003.1.patch > > > Queries like: > select max(ds) from T > where ds is a partitioning column should be optimized. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
