[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715773#comment-13715773 ] Ashutosh Chauhan commented on HIVE-4502: +1 Thanks [~yhuai] for taking a look. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714652#comment-13714652 ] Hive QA commented on HIVE-4502: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12593370/HIVE-4502.patch {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2647 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/117/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/117/console Messages: {noformat} Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714762#comment-13714762 ] Yin Huai commented on HIVE-4502: The failed query in correlationoptimizer3.q is ... {code:sql} set hive.optimize.correlation=true; set hive.auto.convert.join=true; set hive.optimize.mapjoin.mapreduce=true; SELECT d.key AS key, d.cnt AS cnt, b.value as value FROM (SELECT x.key as key, x.value as value FROM src1 x JOIN src y ON (x.key = y.key)) b JOIN (SELECT x.key as key, count(1) AS cnt FROM src1 x JOIN src y ON (x.key = y.key) group by x.key) d ON (b.key = d.key); {\code} and here is the stack trace ... {code} 2013-07-21 10:14:04,440 INFO lazybinary.LazyBinaryStruct (LazyBinaryStruct.java:parse(172)) - Missing fields! Expected 1 fields but only got 0! Ignoring similar problems. 2013-07-21 10:14:04,444 FATAL ExecReducer (ExecReducer.java:reduce(269)) - org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {key:{joinkey0:128},value:{_col1:}} at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:258) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262) Caused by: java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.hadoop.io.Text.set(Text.java:205) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:216) at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:197) at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:61) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236) at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222) at org.apache.hadoop.hive.ql.exec.MuxOperator.processOp(MuxOperator.java:249) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.DemuxOperator.processOp(DemuxOperator.java:245) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504) at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:249) ... 3 more {\code} At here, DemuxOperator only replaces the tag (the tag seems correct at here) and forward the row to MuxOperator. Then, MuxOperator will just forward the row to the JoinOperator. Seems we cannot get the value in this case... Still looking for the root cause... Seems related to MapJoin because the query with hive.optimize.correlation=true and hive.auto.convert.join=false is fine. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714791#comment-13714791 ] Yin Huai commented on HIVE-4502: Problem found. It's because in 'splitTasks' the new patch (uploaded at 20/Jul/13 17:06) did not set needsTagging for the MR job which has DemuxOperator. Will upload a incremental patch later. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714800#comment-13714800 ] Yin Huai commented on HIVE-4502: HIVE-4502.incremental.patch has extra changes I made based on HIVE-4502.patch (uploaded at 20/Jul/13 17:06). Since in GenMapRedUtils there are three places that we need to check if we need to need to do tagging, I add a utility method to check the class of the reducer. We will set needsTagging when the reducer is JoinOperator or DemuxOperator. Also, some small updates are needed for test results of those queries related to Correlation Optimizer. Those updates just replace ' A masked pattern was here ' with '$INTNAME'. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714876#comment-13714876 ] Hive QA commented on HIVE-4502: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12593432/HIVE-4502.1.patch {color:green}SUCCESS:{color} +1 2647 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/122/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/122/console Messages: {noformat} Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714573#comment-13714573 ] Phabricator commented on HIVE-4502: --- ashutoshc has accepted the revision HIVE-4502 [jira] NPE - subquery smb joins fails. +1 REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711534#comment-13711534 ] Phabricator commented on HIVE-4502: --- vikram has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Maybe this can be moved to GenMRProcCtx and a hashmap used there can provide the same functionality. REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711560#comment-13711560 ] Vikram Dixit K commented on HIVE-4502: -- Left some comments on Phabricator. [~navis] could you please take a look. Thanks Vikram. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711561#comment-13711561 ] Phabricator commented on HIVE-4502: --- vikram has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 You could have a hashmap from task to seenOp in the GenMRProCtx that maps a task to its corresponding operator(s). MapTask, ListOperator? should do the trick. This should address Ashutosh's concern. REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712016#comment-13712016 ] Phabricator commented on HIVE-4502: --- navis has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 Ah sorry. This was throwing NPE when debugging. ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Ok, sure. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think it's not possible to have 2 or more children for RS, but might be possible for Tez. I'll do that. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 ok ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 No. Will be changed to package local. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310 MapRedTask in fact. If any of MapRedTask need BucketizedInputFormat, use BucketizedInputFormat. It's fix for https://issues.apache.org/jira/browse/HIVE-4502?focusedCommentId=13679262page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13679262 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 ok. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942 done ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 If Task-A is merged into Task-B, task-A will not be visible anywhere. I've changed not to walk further when all parents are not walked, which changed stage-ids in result. TS1 TS2 SMB RS GBY RS GBY FS before : TS1(stage-1)-SMB-RS-GBY[split]RS(stage-2)-GBY-FS TS2(stage-3)-SMB[merge into stage-1]-RS-GBY-RS-GBY-RS-FS after : TS1(stage-1)-SMB TS2(stage-2)-SMB[merged into stage-1]-RS-GBY[split]RS(stage-3)-GBY-FS I'll make a issue for rearranging Stage-IDs. (and order of explain output, too) ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 It's just a refactoring. setupBucketMapJoinInfo() was called both in initMapJoinPlan() and joinMapJoinPlan(). ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 ok ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 It's called in join-checked block. if (reducer.getClass() == JoinOperator.class) { cplan.setNeedsTagging(true); } REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13710519#comment-13710519 ] Hive QA commented on HIVE-4502: --- {color:green}Overall{color}: +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12592133/HIVE-4502.D10695.3.patch {color:green}SUCCESS:{color} +1 all tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13709352#comment-13709352 ] Phabricator commented on HIVE-4502: --- ashutoshc has requested changes to the revision HIVE-4502 [jira] NPE - subquery smb joins fails. I dont like the idea of tracking operators in the task thereby tasks getting involved in plan manipulation but in absence of alternatives, this may be the way to move forward for now. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 I guess this is unintentional. ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 I am little hesistant about task being aware of operators. Ideally if we have good demarkation of different layers, task (which is sort of a driver to execute a plan) need not be aware of operators contained in its plan. Planner shouldn't wait so late that it needs to track operators in task to generate plan correctly. I don't have a proposal on how to do this better though. At the very least, it will help to add comment explaining the purpose of this list. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think its better to do following: If (op.getNumChild() != 1){ throw new IllegalStateException(Expecting operator + op + to have one child. Found: + op.getNumChild()); } ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 It will be good to add comments on what this method intends to do. Also, I guess you can make this protected, since all the callers of this method are in same package. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 Does this method need to be public? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310 I didnt get this part. Can you add comments why old and curr tasks need to be of type ExecDriver. What about MapredLocalWork? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 Does this method need to be public? Also, please add javadocs for the method. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 throw IllegalStateException in case # of parents != 1 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 I didnt get why did you remove this if check. We shall be needing tagging only for joins. Right? Removing this check implies we will tag always, is that needed? ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942 Better name: hasBranchFinished() ? ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 Why is this function call needed here? Can you please add comment for it? ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 I see that there is no Stage-2 at all. What caused this? Its little weird that we have stages 0,1 and 3 in plan and no stage-2 at all. ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:349 No stage-2 here either. Probably the same cause? REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679315#comment-13679315 ] Phabricator commented on HIVE-4502: --- brock has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. One minor question INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java:265 What do you think of passing the exception into the constructor of the SemanticException instead of printing it to standard error? REVISION DETAIL https://reviews.facebook.net/D10695 To: JIRA, navis Cc: brock NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679321#comment-13679321 ] Phabricator commented on HIVE-4502: --- navis has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java:265 I prefer that way, too. But I've just fixed typo because it's not related to this issue. Can I do that? ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 I should de-comment this. Sorry. REVISION DETAIL https://reviews.facebook.net/D10695 To: JIRA, navis Cc: brock NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678926#comment-13678926 ] Bruce Nelson commented on HIVE-4502: I verified that HIVE-4502-1.patch for HIVE-4650 has been applied to HWorx hive-0.11.0.1.3.0.0-170 and the MapJoin issue in HIVE-4650 still exits. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678929#comment-13678929 ] Bruce Nelson commented on HIVE-4502: Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170 NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678314#comment-13678314 ] Ashutosh Chauhan commented on HIVE-4502: [~navis] Sorry earlier I didnt look at the patch closely. Now, I see you indeed are doing the right thing. We should move forward on this. Did you get a chance to see why results have changed for auto_sortmerge_join_6.q Perhaps, earlier results are wrong ? Also, it will be good to add [~vikram.dixit] 's testcases (or some simplified version of it)in the patch, if its easy enough. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673578#comment-13673578 ] Ashutosh Chauhan commented on HIVE-4502: [~navis] Would you like to take a look at Vikram's patch? I think if we can retain SMBJoin instead of converting them to reduce-side join, thats better. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673856#comment-13673856 ] Navis commented on HIVE-4502: - I've not converted any SMBJoins to RS-joins and just changed creation order of those. The difference is that my patch adds a root task only when all of the join aliases are handled, which is contrary to trunk (add root whenever possible and remove if it's not afterwards). The patch I've attached seemed easier but it is just my call. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673946#comment-13673946 ] Vikram Dixit K commented on HIVE-4502: -- [~navis] I misread the results of the test case from your patch. I was going through your patch more meticulously and found that a few of the tests have different results. Particularly those in auto_sortmerge_join_6.q. The count results seem to have changed. HIVE-3891 converts SMB joins to map-joins when possible. Although that seems orthogonal to this change, any idea as to why the join is still SMB? Also attached a few more tests for this. The plans seem valid after applying your patch. I will continue to review the patch. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662539#comment-13662539 ] Vikram Dixit K commented on HIVE-4502: -- Review board request: https://reviews.apache.org/r/11082/ NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira