[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715773#comment-13715773 ] Ashutosh Chauhan commented on HIVE-4502: +1 Thanks [~yhuai] for taking a look. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714652#comment-13714652
]
Hive QA commented on HIVE-4502:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593370/HIVE-4502.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2647 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/117/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714762#comment-13714762
]
Yin Huai commented on HIVE-4502:
The failed query in correlationoptimizer3.q is ...
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT d.key AS key, d.cnt AS cnt, b.value as value
FROM (SELECT x.key as key, x.value as value FROM src1 x JOIN src y ON (x.key =
y.key)) b
JOIN (SELECT x.key as key, count(1) AS cnt FROM src1 x JOIN src y ON (x.key =
y.key) group by x.key) d
ON (b.key = d.key);
{\code}
and here is the stack trace ...
{code}
2013-07-21 10:14:04,440 INFO lazybinary.LazyBinaryStruct
(LazyBinaryStruct.java:parse(172)) - Missing fields! Expected 1 fields but only
got 0! Ignoring similar problems.
2013-07-21 10:14:04,444 FATAL ExecReducer (ExecReducer.java:reduce(269)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0) {key:{joinkey0:128},value:{_col1:}}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:258)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:205)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:216)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:197)
at
org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:61)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
at
org.apache.hadoop.hive.ql.exec.MuxOperator.processOp(MuxOperator.java:249)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.DemuxOperator.processOp(DemuxOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:249)
... 3 more
{\code}
At here, DemuxOperator only replaces the tag (the tag seems correct at here)
and forward the row to MuxOperator. Then, MuxOperator will just forward the row
to the JoinOperator. Seems we cannot get the value in this case... Still
looking for the root cause... Seems related to MapJoin because the query with
hive.optimize.correlation=true and hive.auto.convert.join=false is fine.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714791#comment-13714791 ] Yin Huai commented on HIVE-4502: Problem found. It's because in 'splitTasks' the new patch (uploaded at 20/Jul/13 17:06) did not set needsTagging for the MR job which has DemuxOperator. Will upload a incremental patch later. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714800#comment-13714800 ] Yin Huai commented on HIVE-4502: HIVE-4502.incremental.patch has extra changes I made based on HIVE-4502.patch (uploaded at 20/Jul/13 17:06). Since in GenMapRedUtils there are three places that we need to check if we need to need to do tagging, I add a utility method to check the class of the reducer. We will set needsTagging when the reducer is JoinOperator or DemuxOperator. Also, some small updates are needed for test results of those queries related to Correlation Optimizer. Those updates just replace ' A masked pattern was here ' with '$INTNAME'. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714876#comment-13714876
]
Hive QA commented on HIVE-4502:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593432/HIVE-4502.1.patch
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/122/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch,
HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch,
HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q,
smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714573#comment-13714573 ] Phabricator commented on HIVE-4502: --- ashutoshc has accepted the revision HIVE-4502 [jira] NPE - subquery smb joins fails. +1 REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711534#comment-13711534 ] Phabricator commented on HIVE-4502: --- vikram has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Maybe this can be moved to GenMRProcCtx and a hashmap used there can provide the same functionality. REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711560#comment-13711560 ] Vikram Dixit K commented on HIVE-4502: -- Left some comments on Phabricator. [~navis] could you please take a look. Thanks Vikram. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711561#comment-13711561 ] Phabricator commented on HIVE-4502: --- vikram has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 You could have a hashmap from task to seenOp in the GenMRProCtx that maps a task to its corresponding operator(s). MapTask, ListOperator? should do the trick. This should address Ashutosh's concern. REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712016#comment-13712016
]
Phabricator commented on HIVE-4502:
---
navis has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins
fails.
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 Ah sorry. This was
throwing NPE when debugging.
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 Ok, sure.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think
it's not possible to have 2 or more children for RS, but might be possible for
Tez. I'll do that.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 ok
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 No.
Will be changed to package local.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310
MapRedTask in fact. If any of MapRedTask need BucketizedInputFormat, use
BucketizedInputFormat. It's fix for
https://issues.apache.org/jira/browse/HIVE-4502?focusedCommentId=13679262page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13679262
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 ok.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942 done
ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 If Task-A is
merged into Task-B, task-A will not be visible anywhere. I've changed not to
walk further when all parents are not walked, which changed stage-ids in result.
TS1 TS2
SMB
RS
GBY
RS
GBY
FS
before :
TS1(stage-1)-SMB-RS-GBY[split]RS(stage-2)-GBY-FS
TS2(stage-3)-SMB[merge into stage-1]-RS-GBY-RS-GBY-RS-FS
after :
TS1(stage-1)-SMB
TS2(stage-2)-SMB[merged into stage-1]-RS-GBY[split]RS(stage-3)-GBY-FS
I'll make a issue for rearranging Stage-IDs. (and order of explain output,
too)
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 It's
just a refactoring. setupBucketMapJoinInfo() was called both in
initMapJoinPlan() and joinMapJoinPlan().
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 ok
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 It's
called in join-checked block.
if (reducer.getClass() == JoinOperator.class) {
cplan.setNeedsTagging(true);
}
REVISION DETAIL
https://reviews.facebook.net/D10695
BRANCH
HIVE-4502
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: brock, vikram
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13710519#comment-13710519
]
Hive QA commented on HIVE-4502:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12592133/HIVE-4502.D10695.3.patch
{color:green}SUCCESS:{color} +1 all tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8/console
Messages:
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
This message is automatically generated.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13709352#comment-13709352
]
Phabricator commented on HIVE-4502:
---
ashutoshc has requested changes to the revision HIVE-4502 [jira] NPE -
subquery smb joins fails.
I dont like the idea of tracking operators in the task thereby tasks getting
involved in plan manipulation but in absence of alternatives, this may be the
way to move forward for now.
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 I guess this is
unintentional.
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:72 I am little hesistant
about task being aware of operators. Ideally if we have good demarkation of
different layers, task (which is sort of a driver to execute a plan) need not
be aware of operators contained in its plan. Planner shouldn't wait so late
that it needs to track operators in task to generate plan correctly. I don't
have a proposal on how to do this better though. At the very least, it will
help to add comment explaining the purpose of this list.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRRedSink1.java:70 I think
its better to do following:
If (op.getNumChild() != 1){
throw new IllegalStateException(Expecting operator + op + to have one
child. Found: + op.getNumChild());
}
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:261 It
will be good to add comments on what this method intends to do. Also, I guess
you can make this protected, since all the callers of this method are in same
package.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:281 Does
this method need to be public?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:310 I
didnt get this part. Can you add comments why old and curr tasks need to be of
type ExecDriver. What about MapredLocalWork?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:318 Does
this method need to be public? Also, please add javadocs for the method.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:851 throw
IllegalStateException in case # of parents != 1
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:949 I
didnt get why did you remove this if check. We shall be needing tagging only
for joins. Right? Removing this check implies we will tag always, is that
needed?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:942
Better name: hasBranchFinished() ?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinFactory.java:228 Why
is this function call needed here? Can you please add comment for it?
ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:165 I see that
there is no Stage-2 at all. What caused this? Its little weird that we have
stages 0,1 and 3 in plan and no stage-2 at all.
ql/src/test/results/clientpositive/auto_smb_mapjoin_14.q.out:349 No stage-2
here either. Probably the same cause?
REVISION DETAIL
https://reviews.facebook.net/D10695
BRANCH
HIVE-4502
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, navis
Cc: brock
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679315#comment-13679315 ] Phabricator commented on HIVE-4502: --- brock has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. One minor question INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java:265 What do you think of passing the exception into the constructor of the SemanticException instead of printing it to standard error? REVISION DETAIL https://reviews.facebook.net/D10695 To: JIRA, navis Cc: brock NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679321#comment-13679321 ] Phabricator commented on HIVE-4502: --- navis has commented on the revision HIVE-4502 [jira] NPE - subquery smb joins fails. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java:265 I prefer that way, too. But I've just fixed typo because it's not related to this issue. Can I do that? ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:619 I should de-comment this. Sorry. REVISION DETAIL https://reviews.facebook.net/D10695 To: JIRA, navis Cc: brock NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678926#comment-13678926 ] Bruce Nelson commented on HIVE-4502: I verified that HIVE-4502-1.patch for HIVE-4650 has been applied to HWorx hive-0.11.0.1.3.0.0-170 and the MapJoin issue in HIVE-4650 still exits. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678929#comment-13678929 ] Bruce Nelson commented on HIVE-4502: Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170 NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678314#comment-13678314 ] Ashutosh Chauhan commented on HIVE-4502: [~navis] Sorry earlier I didnt look at the patch closely. Now, I see you indeed are doing the right thing. We should move forward on this. Did you get a chance to see why results have changed for auto_sortmerge_join_6.q Perhaps, earlier results are wrong ? Also, it will be good to add [~vikram.dixit] 's testcases (or some simplified version of it)in the patch, if its easy enough. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673578#comment-13673578 ] Ashutosh Chauhan commented on HIVE-4502: [~navis] Would you like to take a look at Vikram's patch? I think if we can retain SMBJoin instead of converting them to reduce-side join, thats better. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673856#comment-13673856 ] Navis commented on HIVE-4502: - I've not converted any SMBJoins to RS-joins and just changed creation order of those. The difference is that my patch adds a root task only when all of the join aliases are handled, which is contrary to trunk (add root whenever possible and remove if it's not afterwards). The patch I've attached seemed easier but it is just my call. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673946#comment-13673946 ] Vikram Dixit K commented on HIVE-4502: -- [~navis] I misread the results of the test case from your patch. I was going through your patch more meticulously and found that a few of the tests have different results. Particularly those in auto_sortmerge_join_6.q. The count results seem to have changed. HIVE-3891 converts SMB joins to map-joins when possible. Although that seems orthogonal to this change, any idea as to why the join is still SMB? Also attached a few more tests for this. The plans seem valid after applying your patch. I will continue to review the patch. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662539#comment-13662539 ] Vikram Dixit K commented on HIVE-4502: -- Review board request: https://reviews.apache.org/r/11082/ NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
