[ https://issues.apache.org/jira/browse/DRILL-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973736#comment-14973736 ]
ASF GitHub Bot commented on DRILL-3975: --------------------------------------- Github user StevenMPhillips commented on the pull request: https://github.com/apache/drill/pull/217#issuecomment-151026505 +1 > Partition Planning rule causes query failure due to IndexOutOfBoundsException > on HDFS > ------------------------------------------------------------------------------------- > > Key: DRILL-3975 > URL: https://issues.apache.org/jira/browse/DRILL-3975 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization > Reporter: Jacques Nadeau > Assignee: Steven Phillips > > In attempting to run the extended test suite provided by MapR, there are a > large number of queries that fail due to issues in the PruneScanRule and > specifically the DFSPartitionLocation constructor line 31. It is likely due > to issues with the code that are related to running on HDFS where this code > path has apparently not been tested. > An example test query this type of failure occurred: > /src/drill-test-framework/resources/Functional/ctas/ctas_auto_partition/tpch0.01_multiple_partitions/data/q11.q > Example stack trace below: > {code} > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > StringIndexOutOfBoundsException: String index out of range: -12 > [Error Id: f2941267-49b1-4f67-a17f-610ffb13fcb7 on > ip-172-31-30-32.us-west-2.compute.internal:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) > ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) > [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_85] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_85] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85] > Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected > exception during fragment initialization: Internal error: Error while > applying rule PruneScanRule:Filter_On_Scan_Parquet, args > [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0, > 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, > ctasAutoPartition, > tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan > [entries=[ReadEntryWithPath > [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]], > > selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2, > numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])] > ... 4 common frames omitted > Caused by: java.lang.AssertionError: Internal error: Error while applying > rule PruneScanRule:Filter_On_Scan_Parquet, args > [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0, > 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, > ctasAutoPartition, > tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan > [entries=[ReadEntryWithPath > [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]], > > selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2, > numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])] > at org.apache.calcite.util.Util.newInternal(Util.java:792) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) > [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > ... 3 common frames omitted > Caused by: java.lang.StringIndexOutOfBoundsException: String index out of > range: -12 > at java.lang.String.substring(String.java:1875) ~[na:1.7.0_85] > at > org.apache.drill.exec.planner.DFSPartitionLocation.<init>(DFSPartitionLocation.java:31) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.ParquetPartitionDescriptor.createPartitionSublists(ParquetPartitionDescriptor.java:126) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.AbstractPartitionDescriptor.iterator(AbstractPartitionDescriptor.java:53) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:190) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87) > ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) > ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6] > ... 13 common frames omitted > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)