[
https://issues.apache.org/jira/browse/DRILL-8511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883885#comment-17883885
]
ASF GitHub Bot commented on DRILL-8511:
---------------------------------------
rymarm commented on PR #2943:
URL: https://github.com/apache/drill/pull/2943#issuecomment-2368315909
@paul-rogers thank you so much for the review! I found and fixed one mistake
which the failed tests highlighted already and will write additional tests for
the cases you described.
I agree that test-driven development is much safer and better than a simple
fix without test coverage of the affected place. It is just hard to me to write
tests for Drill, they looks complicated and not isolated to me. But I'll write
them.
> Overflow appeared when the batch reached rows limit
> ---------------------------------------------------
>
> Key: DRILL-8511
> URL: https://issues.apache.org/jira/browse/DRILL-8511
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.21.2
> Reporter: Maksym Rymar
> Assignee: Maksym Rymar
> Priority: Major
> Attachments: complex.zip
>
>
>
> Drill fails to read a JSON file with the exception:
> {{java.lang.IllegalStateException: Unexpected state: FULL_BATCH:}}
> {code:java}
> Caused by: java.lang.IllegalStateException: Unexpected state: FULL_BATCH
> at
> org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.overflowed(ResultSetLoaderImpl.java:639)
> at
> org.apache.drill.exec.physical.resultSet.impl.ColumnState$PrimitiveColumnState.overflowed(ColumnState.java:73)
> at
> org.apache.drill.exec.vector.accessor.writer.BaseScalarWriter.overflowed(BaseScalarWriter.java:214)
> at
> org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.resize(AbstractFixedWidthWriter.java:249)
> at
> org.apache.drill.exec.vector.accessor.writer.BitColumnWriter.prepareWrite(BitColumnWriter.java:77)
> at
> org.apache.drill.exec.vector.accessor.writer.BitColumnWriter.setValueCount(BitColumnWriter.java:87)
> at
> org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.endWrite(AbstractFixedWidthWriter.java:299)
> at
> org.apache.drill.exec.vector.accessor.writer.NullableScalarWriter.endWrite(NullableScalarWriter.java:298)
> at
> org.apache.drill.exec.vector.accessor.writer.AbstractTupleWriter.endWrite(AbstractTupleWriter.java:366)
> at
> org.apache.drill.exec.physical.resultSet.impl.RowSetLoaderImpl.endBatch(RowSetLoaderImpl.java:101)
> at
> org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.harvestNormalBatch(ResultSetLoaderImpl.java:730)
> at
> org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.harvest(ResultSetLoaderImpl.java:700)
> at
> org.apache.drill.exec.physical.impl.scan.project.ReaderSchemaOrchestrator.endBatch(ReaderSchemaOrchestrator.java:137)
> at
> org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.next(ShimBatchReader.java:148)
> at
> org.apache.drill.exec.physical.impl.scan.ReaderState.readBatch(ReaderState.java:400)
> at
> org.apache.drill.exec.physical.impl.scan.ReaderState.next(ReaderState.java:361)
> at
> org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction(ScanOperatorExec.java:270)
> at
> org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next(ScanOperatorExec.java:242)
> at
> org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext(OperatorDriver.java:201)
> at
> org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start(OperatorDriver.java:179)
> at
> org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next(OperatorDriver.java:129)
> at
> org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next(OperatorRecordBatch.java:149)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101)
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:59)
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:93)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:161)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103)
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0(FragmentExecutor.java:324)
> at .......(:0)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2012)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at .......(:0) {code}
> Overflow appeared when the batch reached the rows limit with JSON reader.
> To reproduce the issue - execute the following query against the attached
> file:
>
> {code:java}
> SELECT id,
> gbyi,
> gbyt,
> fl,
> nul,
> bool,
> str,
> sia,
> sfa,
> soa,
> ooa,
> oooi,
> ooof,
> ooos,
> oooa
> FROM dfs.tmp.`complex.json` {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
