[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633058#comment-13633058 ] Hudson commented on HIVE-3996: -- Integrated in Hive-trunk-hadoop2 #162 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/162/]) HIVE-3996 Correctly enforce the memory limit on the multi-table map-join (Vikram Dixit via namit) (Revision 1468321) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1468321 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java * /hive/trunk/ql/src/test/queries/clientpositive/join32_lessSize.q * /hive/trunk/ql/src/test/results/clientpositive/join32_lessSize.q.out Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996_7.patch, HIVE-3996_8.patch, HIVE-3996_9.patch, hive.3996.9.patch-nohcat, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633657#comment-13633657 ] Hudson commented on HIVE-3996: -- Integrated in Hive-trunk-h0.21 #2067 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2067/]) HIVE-3996 Correctly enforce the memory limit on the multi-table map-join (Vikram Dixit via namit) (Revision 1468321) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1468321 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java * /hive/trunk/ql/src/test/queries/clientpositive/join32_lessSize.q * /hive/trunk/ql/src/test/results/clientpositive/join32_lessSize.q.out Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996_7.patch, HIVE-3996_8.patch, HIVE-3996_9.patch, hive.3996.9.patch-nohcat, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629969#comment-13629969 ] Namit Jain commented on HIVE-3996: -- +1 Running tests Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996_7.patch, HIVE-3996_8.patch, HIVE-3996_9.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623287#comment-13623287 ] Vikram Dixit K commented on HIVE-3996: -- Comments addressed. [~namit] please let me know. Thanks Vikram. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996_7.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620244#comment-13620244 ] Ashutosh Chauhan commented on HIVE-3996: [~vikram.dixit] Can you also update your RB entry with your latest patch? Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620249#comment-13620249 ] Vikram Dixit K commented on HIVE-3996: -- https://reviews.apache.org/r/9358/ Thanks Vikram. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620289#comment-13620289 ] Ashutosh Chauhan commented on HIVE-3996: +1 will commit if tests pass. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620579#comment-13620579 ] Namit Jain commented on HIVE-3996: -- [~ashutoshc], can you hold on ? I wanted to take a look at this one more time. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996_3.patch, HIVE-3996_4.patch, HIVE-3996_5.patch, HIVE-3996_6.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580424#comment-13580424 ] Namit Jain commented on HIVE-3996: -- There is a bug in the patch. For sub-queries, we cannot figure out the size at compile time - for eg. join29.q. It should have a backup task. In the new function, getBigTablePosition, you are loosing the variable convertMapJoin, since it is a local variable. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580429#comment-13580429 ] Namit Jain commented on HIVE-3996: -- Also, the new results for union22.q are wrong. What is the problem ? Essentially, in the current code, the nocondtionaltask is not enforced for a sub-query. Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577814#comment-13577814 ] Vikram Dixit K commented on HIVE-3996: -- https://reviews.apache.org/r/9429/ Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996_2.patch, HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573820#comment-13573820 ] Vikram Dixit K commented on HIVE-3996: -- Review board entry: https://reviews.apache.org/r/9358/ Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3996) Correctly enforce the memory limit on the multi-table map-join
[ https://issues.apache.org/jira/browse/HIVE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573156#comment-13573156 ] Namit Jain commented on HIVE-3996: -- Can you create a phabricator entry ? Correctly enforce the memory limit on the multi-table map-join -- Key: HIVE-3996 URL: https://issues.apache.org/jira/browse/HIVE-3996 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-3996.patch Currently with HIVE-3784, the joins are converted to map-joins based on checks of the table size against the config variable: hive.auto.convert.join.noconditionaltask.size. However, the current implementation will also merge multiple mapjoin operators into a single task regardless of whether the sum of the table sizes will exceed the configured value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
