[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-10163: -- Fix Version/s: 1.2.0 CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gunther Hagleitner Labels: JOIN, Performance Fix For: 1.2.0 Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, HIVE-10163.3.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, mergejoin-parallel-lock.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-parallel-lock.png! !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: (was: HIVE-10163.2.patch) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gunther Hagleitner Labels: JOIN, Performance Attachments: HIVE-10163.1.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, mergejoin-parallel-lock.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-parallel-lock.png! !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: HIVE-10163.2.patch CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gunther Hagleitner Labels: JOIN, Performance Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, mergejoin-parallel-lock.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-parallel-lock.png! !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: HIVE-10163.2.patch [~hagleitn]: The thread locking disappeared with your patch, but I found the Hashmap in a few traces, which is pointless when maxAlias is bounded. Also fixed the JIT performance for the 1-key JOIN clauses cross-products. CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gunther Hagleitner Labels: JOIN, Performance Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, mergejoin-parallel-lock.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-parallel-lock.png! !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: mergejoin-parallel-lock.png mergejoin-parallel-bt.png CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Labels: JOIN, Performance Attachments: mergejoin-comparekeys.png, mergejoin-parallel-bt.png, mergejoin-parallel-lock.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Description: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. was: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Labels: JOIN, Performance Attachments: mergejoin-comparekeys.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: mergejoin-comparekeys.png CommonMergeJoinOperator calls WritableComparator.get() in the inner loop Key: HIVE-10163 URL: https://issues.apache.org/jira/browse/HIVE-10163 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.0 Reporter: Gopal V Labels: JOIN, Performance Attachments: mergejoin-comparekeys.png The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings(rawtypes) private int compareKeys(ListObject k1, ListObject k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)