[
https://issues.apache.org/jira/browse/DRILL-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972998#comment-14972998
]
Mehant Baid commented on DRILL-3975:
------------------------------------
This particular bug was happening when ParquetPruneScanRule was hitting IOOB in
the logic you pointed out when there was no need to perform any splitting
(since for auto partitioning scheme we get the partitioning column value from
the file and not from the location).
However while debugging this I found that the "selectionRoot" contained the
scheme and "file" did not contain the scheme potentially causing IOOB you might
be seeing. Stripping out the scheme makes sense, we cannot check for -1
[Here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSPartitionLocation.java#L30]
as it would cause the partitioning columns to be incorrectly empty.
> Partition Planning rule causes query failure due to IndexOutOfBoundsException
> on HDFS
> -------------------------------------------------------------------------------------
>
> Key: DRILL-3975
> URL: https://issues.apache.org/jira/browse/DRILL-3975
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Reporter: Jacques Nadeau
>
> In attempting to run the extended test suite provided by MapR, there are a
> large number of queries that fail due to issues in the PruneScanRule and
> specifically the DFSPartitionLocation constructor line 31. It is likely due
> to issues with the code that are related to running on HDFS where this code
> path has apparently not been tested.
> An example test query this type of failure occurred:
> /src/drill-test-framework/resources/Functional/ctas/ctas_auto_partition/tpch0.01_multiple_partitions/data/q11.q
> Example stack trace below:
> {code}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> StringIndexOutOfBoundsException: String index out of range: -12
> [Error Id: f2941267-49b1-4f67-a17f-610ffb13fcb7 on
> ip-172-31-30-32.us-west-2.compute.internal:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_85]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_85]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: Error while
> applying rule PruneScanRule:Filter_On_Scan_Parquet, args
> [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0,
> 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs,
> ctasAutoPartition,
> tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]],
>
> selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2,
> numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule PruneScanRule:Filter_On_Scan_Parquet, args
> [rel#43148:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#43147:Subset#4.LOGICAL.ANY([]).[],condition==($0,
> 1)), rel#43241:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs,
> ctasAutoPartition,
> tpch_multiple_partitions/lineitem_twopart_ordered2],groupscan=ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2]],
>
> selectionRoot=hdfs://ip-172-31-30-32:54310/drill/testdata/ctas_auto_partition/tpch_multiple_partitions/lineitem_twopart_ordered2,
> numFiles=1, usedMetadataFile=false, columns=[`l_modline`, `l_moddate`]])]
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -12
> at java.lang.String.substring(String.java:1875) ~[na:1.7.0_85]
> at
> org.apache.drill.exec.planner.DFSPartitionLocation.<init>(DFSPartitionLocation.java:31)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.ParquetPartitionDescriptor.createPartitionSublists(ParquetPartitionDescriptor.java:126)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.AbstractPartitionDescriptor.iterator(AbstractPartitionDescriptor.java:53)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:190)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
> ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> ... 13 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
