[jira] [Updated] (HIVE-4968) When deduplicate multiple SelectOperators, we should update RowResolver accordinly
[ https://issues.apache.org/jira/browse/HIVE-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4968: --- Summary: When deduplicate multiple SelectOperators, we should update RowResolver accordinly (was: Broken plan in MapJoin) > When deduplicate multiple SelectOperators, we should update RowResolver > accordinly > -- > > Key: HIVE-4968 > URL: https://issues.apache.org/jira/browse/HIVE-4968 > Project: Hive > Issue Type: Bug >Reporter: Yin Huai >Assignee: Yin Huai > > {code:Sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT key, value > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code:sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT * > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code} > SELECT tmp4.key, tmp4.value, tmp4.count > FROM (SELECT tmp2.key as key, tmp2.value as value, tmp3.count as count > FROM (SELECT * > FROM (SELECT key, value > FROM src) tmp1 ) tmp2 > JOIN (SELECT count(*) as count > FROM src) tmp3 > ) tmp4; > {\code} > The plan is not executable. > The plan related to the MapJoin is > {code} > Stage: Stage-5 > Map Reduce Local Work > Alias -> Map Local Tables: > tmp4:tmp2:tmp1:src > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > tmp4:tmp2:tmp1:src > TableScan > alias: src > Select Operator > expressions: > expr: key > type: string > expr: value > type: string > outputColumnNames: _col0, _col1 > HashTable Sink Operator > condition expressions: > 0 > 1 {_col0} > handleSkewJoin: false > keys: > 0 [] > 1 [] > Position of Big Table: 1 > Stage: Stage-4 > Map Reduce > Alias -> Map Operator Tree: > $INTNAME > Map Join Operator > condition map: >Inner Join 0 to 1 > condition expressions: > 0 > 1 {_col0} > handleSkewJoin: false > keys: > 0 [] > 1 [] > outputColumnNames: _col2 > Position of Big Table: 1 > Select Operator > expressions: > expr: _col0 > type: string > expr: _col1 > type: string > expr: _col2 > type: bigint > outputColumnNames: _col0, _col1, _col2 > File Output Operator > compressed: false > GlobalTableId: 0 > table: > input format: org.apache.hadoop.mapred.TextInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > Local Work: > Map Reduce Local Work > {\code} > The outputColumnNames of MapJoin is '_col2'. But it should be '_col0, _col1, > _col2' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4968) When deduplicate multiple SelectOperators, we should update RowResolver accordinly
[ https://issues.apache.org/jira/browse/HIVE-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4968: -- Attachment: HIVE-4968.D11901.1.patch yhuai requested code review of "HIVE-4968 [jira] When deduplicate multiple SelectOperators, we should update RowResolver accordinly". Reviewers: JIRA Merge remote-tracking branch 'origin/trunk' into HIVE-4968 SELECT tmp3.key, tmp3.value, tmp3.count FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count FROM (SELECT key, value FROM src) tmp1 JOIN (SELECT count(*) as count FROM src) tmp2 ) tmp3; The plan is executable. SELECT tmp3.key, tmp3.value, tmp3.count FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count FROM (SELECT * FROM src) tmp1 JOIN (SELECT count(*) as count FROM src) tmp2 ) tmp3; The plan is executable. SELECT tmp4.key, tmp4.value, tmp4.count FROM (SELECT tmp2.key as key, tmp2.value as value, tmp3.count as count FROM (SELECT * FROM (SELECT key, value FROM src) tmp1 ) tmp2 JOIN (SELECT count(*) as count FROM src) tmp3 ) tmp4; The plan is not executable. The plan related to the MapJoin is Stage: Stage-5 Map Reduce Local Work Alias -> Map Local Tables: tmp4:tmp2:tmp1:src Fetch Operator limit: -1 Alias -> Map Local Operator Tree: tmp4:tmp2:tmp1:src TableScan alias: src Select Operator expressions: expr: key type: string expr: value type: string outputColumnNames: _col0, _col1 HashTable Sink Operator condition expressions: 0 1 {_col0} handleSkewJoin: false keys: 0 [] 1 [] Position of Big Table: 1 Stage: Stage-4 Map Reduce Alias -> Map Operator Tree: $INTNAME Map Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 1 {_col0} handleSkewJoin: false keys: 0 [] 1 [] outputColumnNames: _col2 Position of Big Table: 1 Select Operator expressions: expr: _col0 type: string expr: _col1 type: string expr: _col2 type: bigint outputColumnNames: _col0, _col1, _col2 File Output Operator compressed: false GlobalTableId: 0 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Local Work: Map Reduce Local Work The outputColumnNames of MapJoin is '_col2'. But it should be '_col0, _col1, _col2' TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D11901 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java ql/src/test/queries/clientpositive/nonblock_op_deduplicate.q ql/src/test/results/clientpositive/nonblock_op_deduplicate.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/28407/ To: JIRA, yhuai > When deduplicate multiple SelectOperators, we should update RowResolver > accordinly > -- > > Key: HIVE-4968 > URL: https://issues.apache.org/jira/browse/HIVE-4968 > Project: Hive > Issue Type: Bug >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4968.D11901.1.patch > > > {code:Sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT key, value > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code:sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT * > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code} > SELECT tmp4.key, tmp4.value, tmp4.count > FROM (SELECT tmp2.key as key, tmp2.val
[jira] [Updated] (HIVE-4968) When deduplicate multiple SelectOperators, we should update RowResolver accordinly
[ https://issues.apache.org/jira/browse/HIVE-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4968: --- Status: Patch Available (was: Open) > When deduplicate multiple SelectOperators, we should update RowResolver > accordinly > -- > > Key: HIVE-4968 > URL: https://issues.apache.org/jira/browse/HIVE-4968 > Project: Hive > Issue Type: Bug >Reporter: Yin Huai >Assignee: Yin Huai > Attachments: HIVE-4968.D11901.1.patch > > > {code:Sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT key, value > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code:sql} > SELECT tmp3.key, tmp3.value, tmp3.count > FROM (SELECT tmp1.key as key, tmp1.value as value, tmp2.count as count > FROM (SELECT * > FROM src) tmp1 > JOIN (SELECT count(*) as count > FROM src) tmp2 > ) tmp3; > {\code} > The plan is executable. > {code} > SELECT tmp4.key, tmp4.value, tmp4.count > FROM (SELECT tmp2.key as key, tmp2.value as value, tmp3.count as count > FROM (SELECT * > FROM (SELECT key, value > FROM src) tmp1 ) tmp2 > JOIN (SELECT count(*) as count > FROM src) tmp3 > ) tmp4; > {\code} > The plan is not executable. > The plan related to the MapJoin is > {code} > Stage: Stage-5 > Map Reduce Local Work > Alias -> Map Local Tables: > tmp4:tmp2:tmp1:src > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > tmp4:tmp2:tmp1:src > TableScan > alias: src > Select Operator > expressions: > expr: key > type: string > expr: value > type: string > outputColumnNames: _col0, _col1 > HashTable Sink Operator > condition expressions: > 0 > 1 {_col0} > handleSkewJoin: false > keys: > 0 [] > 1 [] > Position of Big Table: 1 > Stage: Stage-4 > Map Reduce > Alias -> Map Operator Tree: > $INTNAME > Map Join Operator > condition map: >Inner Join 0 to 1 > condition expressions: > 0 > 1 {_col0} > handleSkewJoin: false > keys: > 0 [] > 1 [] > outputColumnNames: _col2 > Position of Big Table: 1 > Select Operator > expressions: > expr: _col0 > type: string > expr: _col1 > type: string > expr: _col2 > type: bigint > outputColumnNames: _col0, _col1, _col2 > File Output Operator > compressed: false > GlobalTableId: 0 > table: > input format: org.apache.hadoop.mapred.TextInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > Local Work: > Map Reduce Local Work > {\code} > The outputColumnNames of MapJoin is '_col2'. But it should be '_col0, _col1, > _col2' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira