[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396993#comment-15396993
]
Amareshwari Sriramadasu commented on HIVE-14264:
[~gszadovszky] We are running queries on hive server on version 2.1 and using
MR execution engine. The tables are partitioned and the issue mostly happens
when col is not projected but it is only in filter clause with 'IS NOT NULL'
filter. But we have seen failures in some cases when column is projected as
well.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395685#comment-15395685
]
Gabor Szadovszky commented on HIVE-14264:
-
Are you able to add some details about your environment? Unfortunately, I was
not able to figure anything out of the stacktrace.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395626#comment-15395626
]
Amareshwari Sriramadasu commented on HIVE-14264:
[~gszadovszky], Thanks for looking into this. We were also not able reproduce
in our local setup, query was only failing in our production with above
stacktrace. By disabling cbo, we were able to move forward.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384259#comment-15384259
]
Gabor Szadovszky commented on HIVE-14264:
-
Tried on 2.1.0-rc0 running on Derby db.
Beeline was started using the command: ./beeline --hiveconf
hive.cbo.enable=false -u jdbc:hive2://
Test data created using the following commands:
CREATE DATABASE hive_14264;
USE hive_14264;
CREATE TABLE table1 (key STRING, value STRING);
INSERT INTO TABLE table1
VALUES ('key1', 'value1'), (null, 'value2'), ('key3', null), (null,
null);
Tried to reproduce the issue by using the following queries:
0: jdbc:hive2://> SELECT * FROM table1 WHERE key IS NOT NULL;
OK
+-+---+--+
| table1.key | table1.value |
+-+---+--+
| key1| value1|
| key3| NULL |
+-+---+--+
2 rows selected (0.29 seconds)
0: jdbc:hive2://> SELECT * FROM table1 WHERE value IS NOT NULL;OK
+-+---+--+
| table1.key | table1.value |
+-+---+--+
| key1| value1|
| NULL| value2|
+-+---+--+
2 rows selected (0.087 seconds)
Queries executed as expected; issue was not reproducible.
Could you please provide more info to reproduce the issue?
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
