[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924775#comment-13924775
]
Hive QA commented on HIVE-6403:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633001/HIVE-6403.6.patch.txt
{color:green}SUCCESS:{color} +1 5373 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1655/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1655/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12633001
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, HIVE-6403.5.patch.txt,
> HIVE-6403.6.patch.txt, navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.ha
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904968#comment-13904968
]
Navis commented on HIVE-6403:
-
HIVE-6037 is reverted. I've kicked test.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, HIVE-6403.5.patch.txt,
> navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcess
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904409#comment-13904409
]
Harish Butani commented on HIVE-6403:
-
+1
will checkin once tests pass.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, HIVE-6403.5.patch.txt,
> navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoin
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904072#comment-13904072
]
Hive QA commented on HIVE-6403:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629459/HIVE-6403.5.patch.txt
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1378/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1378/console
Messages:
{noformat}
This message was trimmed, see log for full details
Reverted 'ql/src/test/results/clientpositive/decimal_5.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketmapjoin8.q.out'
Reverted 'ql/src/test/results/clientpositive/progress_1.q.out'
Reverted 'ql/src/test/results/clientpositive/nestedvirtual.q.out'
Reverted 'ql/src/test/results/clientpositive/parallel.q.out'
Reverted 'ql/src/test/results/clientpositive/convert_enum_to_string.q.out'
Reverted 'ql/src/test/results/clientpositive/annotate_stats_filter.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketmapjoin12.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby8_map_skew.q.out'
Reverted 'ql/src/test/results/clientpositive/varchar_1.q.out'
Reverted 'ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out'
Reverted 'ql/src/test/results/clientpositive/join27.q.out'
Reverted
'ql/src/test/results/clientpositive/load_file_with_space_in_the_name.q.out'
Reverted 'ql/src/test/results/clientpositive/load_dyn_part11.q.out'
Reverted 'ql/src/test/results/clientpositive/input4.q.out'
Reverted 'ql/src/test/results/clientpositive/stats6.q.out'
Reverted 'ql/src/test/results/clientpositive/leadlag_queries.q.out'
Reverted 'ql/src/test/results/clientpositive/rename_partition_location.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_translate.q.out'
Reverted 'ql/src/test/results/clientpositive/union12.q.out'
Reverted 'ql/src/test/results/clientpositive/index_compact_2.q.out'
Reverted 'ql/src/test/results/clientpositive/udf2.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out'
Reverted 'ql/src/test/results/clientpositive/windowing_udaf.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_17.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_repeated_alias.q.out'
Reverted 'ql/src/test/results/clientpositive/join36.q.out'
Reverted 'ql/src/test/results/clientpositive/avro_joins.q.out'
Reverted 'ql/src/test/results/clientpositive/drop_with_concurrency.q.out'
Reverted 'ql/src/test/results/clientpositive/lock2.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part5.q.out'
Reverted 'ql/src/test/results/clientpositive/bucketmapjoin3.q.out'
Reverted 'ql/src/test/results/clientpositive/insert_into2.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_skewed_table.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets2.q.out'
Reverted 'ql/src/test/results/clientpositive/timestamp_1.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_vs_table_metadata.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby12.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby1_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/cast1.q.out'
Reverted 'ql/src/test/results/clientpositive/input16_cc.q.out'
Reverted 'ql/src/test/results/clientpositive/annotate_stats_part.q.out'
Reverted 'ql/src/test/results/clientpositive/partition_wise_fileformat15.q.out'
Reverted 'ql/src/test/results/clientpositive/stats1.q.out'
Reverted 'ql/src/test/results/clientpositive/symlink_text_input_format.q.out'
Reverted 'ql/src/test/results/clientpositive/ptf_register_tblfn.q.out'
Reverted 'ql/src/test/results/clientpositive/ddltime.q.out'
Reverted 'ql/src/test/results/clientpositive/uniquejoin.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_constant_expr.q.out'
Reverted 'ql/src/test/results/clientpositive/union30.q.out'
Reverted 'ql/src/test/results/clientpositive/timestamp_null.q.out'
Reverted 'ql/src/test/results/clientpositive/combine1.q.out'
Reverted 'ql/src/test/results/clientpositive/avro_sanity_test.q.out'
Reverted 'ql/src/test/results/clientpositive/exim_06_one_part.q.out'
Reverted
'ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby2_noskew.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale.q.out'
Reverted 'ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out'
Reverted 'ql/src/test/results/clientpositive/smb_mapjoin_12.q.out'
Reverted 'ql/src/test/results/clientpositive/metadataonly1.q.out'
Reverted 'ql/src/test/results/clientpositive/join31.q.out'
Reverted
'ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out'
Reverted
'ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out'
Reverted 'ql/s
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903691#comment-13903691
]
Navis commented on HIVE-6403:
-
It was not that bad as it first looked. I've used task ID for equality of task
but index builder resets ID (even not assigns ID for index metadata updator).
Changed not to reset ID and seemed working good. I'll update the patch shortly
after confirming that.
{noformat}
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
Stage-2 depends on stages: Stage-0
Stage-3 depends on stages: Stage-1
Stage-16 depends on stages: Stage-1, Stage-5, Stage-9, Stage-13
Stage-5 is a root stage
Stage-4 depends on stages: Stage-5
Stage-6 depends on stages: Stage-4
Stage-7 depends on stages: Stage-5
Stage-9 is a root stage
Stage-8 depends on stages: Stage-9
Stage-10 depends on stages: Stage-8
Stage-11 depends on stages: Stage-9
Stage-13 is a root stage
Stage-12 depends on stages: Stage-13
Stage-14 depends on stages: Stage-12
Stage-15 depends on stages: Stage-13
{noformat}
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903674#comment-13903674
]
Navis commented on HIVE-6403:
-
Hive index system seemed broken badly in somewhere.
Even in trunk,
{noformat}
hive> explain extended ALTER INDEX srcpart_index ON srcpart REBUILD;
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
Stage-2 depends on stages: Stage-0
null depends on stages: Stage-1
Stage-3 depends on stages: Stage-1, Stage-1, Stage-1, Stage-1
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
Stage-2 depends on stages: Stage-0
null depends on stages: Stage-1
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
Stage-2 depends on stages: Stage-0
null depends on stages: Stage-1
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
Stage-2 depends on stages: Stage-0
null depends on stages: Stage-1
{noformat}
I've never used index before but it seemed not a valid plan. I'll dig into this
for a while.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.Cl
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903365#comment-13903365
]
Harish Butani commented on HIVE-6403:
-
The patch looks good.
I looked at the failures, most of them are assertion failures because of the
join reorder fix you put in.
There are a few failures to do with Alter Index Rebuild, like index_bitmap_rc.q.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903130#comment-13903130
]
Hive QA commented on HIVE-6403:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12629301/HIVE-6403.4.patch.txt
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 5127 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_update
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1356/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1356/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12629301
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch,
> HIVE-6403.3.patch.txt, HIVE-6403.4.patch.txt, navis.patch, navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Commo
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902912#comment-13902912
]
Navis commented on HIVE-6403:
-
[~rhbutani]
bq. i don't see a union
Ah, it's "auto_join27.q", "not auto_join17.q". Sorry for that.
bq. should favor the right alias as the big table
Seemed a bug which was not fixed properly in HIVE-5945. I'll check that, too.
bq. my contribution is very tiny
I would have never thought of multiInsertBigTableCheck() will be needed. Most
of codes I've suggested are just easier part. I'll make this issue done. Thanks.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch,
> navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901947#comment-13901947
]
Harish Butani commented on HIVE-6403:
-
[~navis] i went through the patch; I think I understand the reason for the
change. Couple of questions:
- i don't see a union e.g. Can an e.g. be added that exercises the map-side
union case.
- i see several plans changed in .out files to scan src1 in place of src2. But
the check at CommonJoinTaskDispatcher::cannotConvert line 447, should favor the
right alias as the big table. So I don't understand why these plans are changed
with your patch. For my understanding, can you please explain this change.
Assigning the jira to you; you have made most of the changes, my contribution
is very tiny; and I am just becoming familiar with the pieces you have touched
in this patch.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch,
> navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901136#comment-13901136
]
Navis commented on HIVE-6403:
-
[~rhbutani] Took some time to understand multiInsertBigTableCheck() and now got
it. But seeing auto_join17.q, I've realized that we missed handling map-side
union cases. Updated in navis2.patch. Could you check that too?
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch,
> navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901088#comment-13901088
]
Harish Butani commented on HIVE-6403:
-
Resubmitting patch. Ran several of the tests locally, they succeed.
The previous run was aborted.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:2
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900810#comment-13900810
]
Hive QA commented on HIVE-6403:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628832/HIVE-6403.2.patch
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 5095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1315/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1315/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12628832
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optim
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900676#comment-13900676
]
Harish Butani commented on HIVE-6403:
-
Ok, yes I see what you mean. I incorporated your findAlias changes. and created
a new patch.
I think the 'multiInsertBigTableCheck' check needs to be still done, because in
case of multi insert we want the 'source' table to be the only big table
choice. Do you agree?
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to genera
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899299#comment-13899299
]
Harish Butani commented on HIVE-6403:
-
What I see is that for the Multi Insert with SubQuery case:
- the first child will be a ReduceSink; other child is converted to FileSink by
the time it gets to CommonJoinTaskDispatcher.
- so for the Multi Insert SubQuery case checking the first child is still ok.
But yes this check that other children are FileSink, should be done in
CommonJoinTaskDispatcher#getPosition
Beyond the above point; I am sorry, I don't follow what else you are proposing.
Can you please elaborate.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at j
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898882#comment-13898882
]
Navis commented on HIVE-6403:
-
I think CommonJoinTaskDispatcher#getPosition() depends on wrong assumption that
top operator has only a child (or resultingly a child like LV cases). We should
track up (child to parent) big aliases from bigTableCandidates, opposed to
current way.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
>
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898852#comment-13898852
]
Harish Butani commented on HIVE-6403:
-
[~navis] could you please review.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897608#comment-13897608
]
Harish Butani commented on HIVE-6403:
-
[~navis] I am no expert on the MapJoinProcessor. Following is what I see; I
will need to spend more time on this.
Maybe from my comments you can see the issue.
1. The Plan generated at genPlan is:
TS[0] (the scan for b) has 2 child operators: [RS[4], RS[25]]
These are for the joins for each of the SubQuery expressions:
b.key in
(select a.key
from src a
where b.value = a.value and a.key > '9'
)
and
b.key not in ( select key from src s1 where s1.key > '2')
The plan looks complex because the handling of not in requires the null check.
This issue will occur even if the second insert is a 'in' subquery predicate.
It will be easier to follow for such an e.g.
2. With set hive.auto.convert.join=false
The second RS gets converted to a FileSink. You can observe this from the
explain output. A subsequent Stage reads this intermediate output to perform
the processing for the 2nd SubQuery.
3. With set hive.auto.convert.join=true;
When it comes to CommonJoinResolver the TS[0] has children [RS[4], FS[44]] ie
the 2nd ReduceSink is converted to a FileSink
The MapJoinProcessor:genMapJoinLocalWork line 145 it is assuming that a
TableScanOp can only have 1 child.
The fix maybe to ignore any FileSink operators that are children of TableScan.
Another test to add is a multi insert on 3 tables.
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Dr
[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true
[
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897467#comment-13897467
]
Navis commented on HIVE-6403:
-
I wish I could fix this but it's way out of my ability. Could you check this
[~rhbutani] ?
> uncorrelated subquery is failing with auto.convert.join=true
>
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working
> with hive.auto.convert.join=true
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b
> > INSERT OVERWRITE TABLE src_4
> > select *
> > where b.key in
> >(select a.key
> > from src a
> > where b.value = a.value and a.key > '9'
> >)
> > INSERT OVERWRITE TABLE src_5
> > select *
> > where b.key not in ( select key from src s1 where s1.key > '2')
> > order by key
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new
> mapJoin operator by exception : Index: 0, Size: 0
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> at
