[ https://issues.apache.org/jira/browse/HIVE-22031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artem Velykorodnyi updated HIVE-22031: -------------------------------------- Attachment: HIVE-22031.02.patch Status: Patch Available (was: Open) > HiveRelDecorrelator fails with IndexOutOfBoundsException if the query > contains several "constant" columns > --------------------------------------------------------------------------------------------------------- > > Key: HIVE-22031 > URL: https://issues.apache.org/jira/browse/HIVE-22031 > Project: Hive > Issue Type: Bug > Components: CBO > Affects Versions: 2.3.5 > Reporter: Artem Velykorodnyi > Assignee: Artem Velykorodnyi > Priority: Major > Attachments: HIVE-22031.02.patch, HIVE-22031.1.patch, HIVE-22031.patch > > > Steps for reproducing: > {code} > 1. Create table orders > create table orders (ORD_NUM INT, CUST_CODE STRING); > 2. Create table customers > create table customers (CUST_CODE STRING); > 3. Make select with constants and with a subquery: > select DISTINCT(CUST_CODE), '777' as ANY, ORD_NUM, '888' as CONSTANT > from orders > WHERE not exists > (select 1 > from customers > WHERE CUST_CODE=orders.CUST_CODE > ); > {code} > Query fails with IndexOutOfBoundsException > {code} > Exception in thread "main" java.lang.AssertionError: Internal error: While > invoking method 'public > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject) > throws org.apache.hadoop.hive.ql.parse.SemanticException' > at org.apache.calcite.util.Util.newInternal(Util.java:792) > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelate(HiveRelDecorrelator.java:252) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateQuery(HiveRelDecorrelator.java:218) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1347) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261) > at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113) > at > org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997) > at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149) > at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:233) > at org.apache.hadoop.util.RunJar.main(RunJar.java:148) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531) > ... 32 more > Caused by: java.lang.AssertionError: Internal error: While invoking method > 'public > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate) > throws org.apache.hadoop.hive.ql.parse.SemanticException' > at org.apache.calcite.util.Util.newInternal(Util.java:792) > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(HiveRelDecorrelator.java:854) > ... 37 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531) > ... 39 more > Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 2 > at java.util.ArrayList.rangeCheckForAdd(ArrayList.java:665) > at java.util.ArrayList.add(ArrayList.java:477) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(HiveRelDecorrelator.java:833) > ... 44 more > {code} > HiveRelDecorrelator looking for omitted constants and put them into TreeMap > where keys are the numbers of columns in top-level select query. > For query from example TreeMap contains: > {code} > 0 = {TreeMap$Entry@8389} "1" -> "_UTF-16LE'777'" > 1 = {TreeMap$Entry@8390} "3" -> "_UTF-16LE'888'" > {code} > After that, there is step where List of fields is combined with contsants > from TreeMap > {code} > if (!omittedConstants.isEmpty()) { > final List<RexNode> postProjects = new ArrayList<>(relBuilder.fields()); > for (Map.Entry<Integer, RexLiteral> entry > : omittedConstants.descendingMap().entrySet()) { > postProjects.add(entry.getKey() + frame.corDefOutputs.size(), > entry.getValue()); > } > relBuilder.project(postProjects); > } > {code} > But TreeMap is descending, so firstly goes constant columns with high > position number, greater than target List size. > (For query from example there is an attempt to add an element to the List > with index 3, but the size of List is only 2). > If we use TreeMap without descending - everything goes as expected. Also, > there is no difference between descending and ascending map, because the List > is filled using indexes but not sequential position. > "Q file" with the query from the example works fine but fails on the Hive > 2.3.5. -- This message was sent by Atlassian JIRA (v7.6.14#76016)