Maksym Rymar created DRILL-8511:
-----------------------------------
Summary: Overflow appeared when the batch reached rows limit
Key: DRILL-8511
URL: https://issues.apache.org/jira/browse/DRILL-8511
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.21.2
Reporter: Maksym Rymar
Assignee: Maksym Rymar
Attachments: complex.zip
Drill fails to read a JSON file with the exception:
{{java.lang.IllegalStateException: Unexpected state: FULL_BATCH:}}
{code:java}
Caused by: java.lang.IllegalStateException: Unexpected state: FULL_BATCH
at
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.overflowed(ResultSetLoaderImpl.java:639)
at
org.apache.drill.exec.physical.resultSet.impl.ColumnState$PrimitiveColumnState.overflowed(ColumnState.java:73)
at
org.apache.drill.exec.vector.accessor.writer.BaseScalarWriter.overflowed(BaseScalarWriter.java:214)
at
org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.resize(AbstractFixedWidthWriter.java:249)
at
org.apache.drill.exec.vector.accessor.writer.BitColumnWriter.prepareWrite(BitColumnWriter.java:77)
at
org.apache.drill.exec.vector.accessor.writer.BitColumnWriter.setValueCount(BitColumnWriter.java:87)
at
org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.endWrite(AbstractFixedWidthWriter.java:299)
at
org.apache.drill.exec.vector.accessor.writer.NullableScalarWriter.endWrite(NullableScalarWriter.java:298)
at
org.apache.drill.exec.vector.accessor.writer.AbstractTupleWriter.endWrite(AbstractTupleWriter.java:366)
at
org.apache.drill.exec.physical.resultSet.impl.RowSetLoaderImpl.endBatch(RowSetLoaderImpl.java:101)
at
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.harvestNormalBatch(ResultSetLoaderImpl.java:730)
at
org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.harvest(ResultSetLoaderImpl.java:700)
at
org.apache.drill.exec.physical.impl.scan.project.ReaderSchemaOrchestrator.endBatch(ReaderSchemaOrchestrator.java:137)
at
org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.next(ShimBatchReader.java:148)
at
org.apache.drill.exec.physical.impl.scan.ReaderState.readBatch(ReaderState.java:400)
at
org.apache.drill.exec.physical.impl.scan.ReaderState.next(ReaderState.java:361)
at
org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction(ScanOperatorExec.java:270)
at
org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next(ScanOperatorExec.java:242)
at
org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext(OperatorDriver.java:201)
at
org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start(OperatorDriver.java:179)
at
org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next(OperatorDriver.java:129)
at
org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next(OperatorRecordBatch.java:149)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101)
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:59)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:93)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:161)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0(FragmentExecutor.java:324)
at .......(:0)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2012)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at .......(:0) {code}
Overflow appeared when the batch reached the rows limit with JSON reader.
To reproduce the issue - execute the following query against the attached file:
{code:java}
SELECT id,
gbyi,
gbyt,
fl,
nul,
bool,
str,
sia,
sfa,
soa,
ooa,
oooi,
ooof,
ooos,
oooa
FROM dfs.tmp.`complex.json` {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
