[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-10163: -- Fix Version/s: 1.2.0 > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Fix For: 1.2.0 > > Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, > HIVE-10163.3.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-10163: -- Attachment: HIVE-10163.3.patch > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, > HIVE-10163.3.patch, mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: HIVE-10163.2.patch > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, > mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: (was: HIVE-10163.2.patch) > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Attachments: HIVE-10163.1.patch, mergejoin-comparekeys.png, > mergejoin-parallel-bt.png, mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: HIVE-10163.2.patch [~hagleitn]: The thread locking disappeared with your patch, but I found the Hashmap in a few traces, which is pointless when maxAlias is bounded. Also fixed the JIT performance for the 1-key JOIN clauses & cross-products. > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Attachments: HIVE-10163.1.patch, HIVE-10163.2.patch, > mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-10163: -- Attachment: HIVE-10163.1.patch > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V >Assignee: Gunther Hagleitner > Labels: JOIN, Performance > Attachments: HIVE-10163.1.patch, mergejoin-comparekeys.png, > mergejoin-parallel-bt.png, mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Description: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings("rawtypes") private int compareKeys(List k1, List k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-parallel-lock.png! !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. !mergejoin-parallel-bt.png! was: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings("rawtypes") private int compareKeys(List k1, List k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V > Labels: JOIN, Performance > Attachments: mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-parallel-lock.png! > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. > !mergejoin-parallel-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: mergejoin-parallel-lock.png mergejoin-parallel-bt.png > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V > Labels: JOIN, Performance > Attachments: mergejoin-comparekeys.png, mergejoin-parallel-bt.png, > mergejoin-parallel-lock.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Description: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings("rawtypes") private int compareKeys(List k1, List k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} !mergejoin-comparekeys.png! The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. was: The CommonMergeJoinOperator wastes CPU looking up the correct comparator for each WritableComparable in each row. {code} @SuppressWarnings("rawtypes") private int compareKeys(List k1, List k2) { int ret = 0; ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); if (ret != 0) { return ret; } } {code} The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where it tries to use reflection to set the Comparator config for each row being compared. > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V > Labels: JOIN, Performance > Attachments: mergejoin-comparekeys.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > !mergejoin-comparekeys.png! > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10163) CommonMergeJoinOperator calls WritableComparator.get() in the inner loop
[ https://issues.apache.org/jira/browse/HIVE-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-10163: --- Attachment: mergejoin-comparekeys.png > CommonMergeJoinOperator calls WritableComparator.get() in the inner loop > > > Key: HIVE-10163 > URL: https://issues.apache.org/jira/browse/HIVE-10163 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.0 >Reporter: Gopal V > Labels: JOIN, Performance > Attachments: mergejoin-comparekeys.png > > > The CommonMergeJoinOperator wastes CPU looking up the correct comparator for > each WritableComparable in each row. > {code} > @SuppressWarnings("rawtypes") > private int compareKeys(List k1, List k2) { > int ret = 0; > > ret = WritableComparator.get(key_1.getClass()).compare(key_1, key_2); > if (ret != 0) { > return ret; > } > } > {code} > The slow part of that get() is deep within {{ReflectionUtils.setConf}}, where > it tries to use reflection to set the Comparator config for each row being > compared. -- This message was sent by Atlassian JIRA (v6.3.4#6332)