[ https://issues.apache.org/jira/browse/DRILL-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Nadeau updated DRILL-1962: ---------------------------------- Component/s: (was: Execution - Operators) Execution - Data Types > accessing nested array from multiple files causing IndexOutOfBoundException > --------------------------------------------------------------------------- > > Key: DRILL-1962 > URL: https://issues.apache.org/jira/browse/DRILL-1962 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Data Types > Affects Versions: 0.8.0 > Reporter: Chun Chang > Assignee: Parth Chandra > > #Wed Jan 07 18:54:07 EST 2015 > git.commit.id.abbrev=35a350f > If the dataset contains nested array of array, and the data is contained in > more than one file, accessing the second nested array cause > IndexOutOfBoundsException. For example, with the following dataset: > {code} > { > "id": 2, > "oooa": { > "oa": { > "oab": { > "oabc": [ > { > "rowId": 2 > }, > { > "rowValue1": [{"rv1":1, "rv2":2}, {"rva1":3, > "rva2":4}], > "rowValue2": [{"rw1":1, "rw2":2}, {"rwa1":3, > "rwa2":4}] > } > ] > } > } > } > } > {code} > If you put it in two separate files in the same directory, query using wild > card to accessing the two files at the second array level will cause the > exception. > {code} > 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select > t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira*.json` t; > +------------+ > | EXPR$0 | > +------------+ > | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] | > Query failed: Query failed: Failure while running fragment., index: -4, > length: 4 (expected: range(0, 16384)) [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 > on qa-node120.qa.lab:31010 ] > [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 on qa-node120.qa.lab:31010 ] > java.lang.RuntimeException: java.sql.SQLException: Failure while executing > query. > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514) > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148) > at sqlline.SqlLine.print(SqlLine.java:1809) > at sqlline.SqlLine$Commands.execute(SqlLine.java:3766) > at sqlline.SqlLine$Commands.sql(SqlLine.java:3663) > at sqlline.SqlLine.dispatch(SqlLine.java:889) > at sqlline.SqlLine.begin(SqlLine.java:763) > at sqlline.SqlLine.start(SqlLine.java:498) > at sqlline.SqlLine.main(SqlLine.java:460) > 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> > {code} > Same query on single file works. > {code} > 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select > t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira1.json` t; > +------------+ > | EXPR$0 | > +------------+ > | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] | > +------------+ > {code} > stack trace: > {code} > 2015-01-08 14:00:16,127 [2b51020e-daab-a903-ef7c-ef6f9bd606c7:frag:0:0] WARN > o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing > fragment > java.lang.IndexOutOfBoundsException: index: -4, length: 4 (expected: range(0, > 16384)) > at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final] > at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final] > at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final] > at > org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:297) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:542) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setChildrenPosition(RepeatedMapReaderImpl.java:45) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:96) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.test.generated.ProjectorGen4732.doEval(ProjectorTemplate.java:30) > ~[na:na] > at > org.apache.drill.exec.test.generated.ProjectorGen4732.projectRecords(ProjectorTemplate.java:64) > ~[na:na] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:172) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:97) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114) > ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254) > [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > {code} > physical plan: > {code} > 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> explain plan for select > t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira*.json` t; > +------------+------------+ > | text | json | > +------------+------------+ > | 00-00 Screen > 00-01 Project(EXPR$0=[ITEM(ITEM(ITEM(ITEM(ITEM($0, 'oa'), 'oab'), > 'oabc'), 1), 'rowValue1')]) > 00-02 Scan(groupscan=[EasyGroupScan > [selectionRoot=/drill/testdata/complex_type/json/jira2file, numFiles=2, > columns=[`oooa`.`oa`.`oab`.`oabc`[1].`rowValue1`], > files=[maprfs:/drill/testdata/complex_type/json/jira2file/jira1.json, > maprfs:/drill/testdata/complex_type/json/jira2file/jira2.json]]]) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)