Excuse my basic questions, when you say we are you reference Drill coders? So what is Integer.MAX_VALUE bytes? Is that a query time setting? Drillbit setting? Is it editable? How does that value get interpreted for complex data types (objects and arrays).
Not only would the column be helpful, but the source file as well. ( if this is an individual record issue....or is this a cumulative error where the max size of the sum of the length of multiple records of a column is at issue). Thoughts on how as a user I could address this in my dataset? Thanks! On Friday, February 5, 2016, Hanifi Gunes <hgu...@maprtech.com> wrote: > You see this exception because one of the columns in your dataset is larger > than an individual DrillBuf could store. The hard limit > is Integer.MAX_VALUE bytes. Around the time we are trying to expand one of > the buffers, we notice the allocation request is oversized and fail the > query. It would be nice if error message contained the column that raised > this issue though. > > On Fri, Feb 5, 2016 at 1:39 PM, John Omernik <j...@omernik.com > <javascript:;>> wrote: > > > Any thoughts on how to troubleshoot this (I have some fat json data going > > into the buffers apparently) It's not huge data, just wide/complex (total > > size is 1.4 GB) Any thoughts on how to troubleshoot or settings I can > use > > to work through these errors? > > > > > > Thanks! > > > > > > John > > > > > > > > > > Error: SYSTEM ERROR: OversizedAllocationException: Unable to expand the > > buffer. Max allowed buffer size is reached. > > > > > > > > Fragment 1:11 > > > > > > > > [Error Id: db21dea0-ddd7-4fcf-9fea-b5031e358dad on node1 > > > > > > > > (org.apache.drill.exec.exception.OversizedAllocationException) Unable > to > > expand the buffer. Max allowed buffer size is reached. > > > > org.apache.drill.exec.vector.UInt1Vector.reAlloc():214 > > > > org.apache.drill.exec.vector.UInt1Vector$Mutator.setValueCount():469 > > > > > > > org.apache.drill.exec.vector.complex.ListVector$Mutator.setValueCount():324 > > > > org.apache.drill.exec.physical.impl.ScanBatch.next():247 > > > > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > > > > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > > > > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51 > > > > > > > > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132 > > > > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > > > > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > > > > > > > > > org.apache.drill.exec.test.generated.StreamingAggregatorGen1931.doWork():172 > > > > > > > > > org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():167 > > > > org.apache.drill.exec.record.AbstractRecordBatch.next():162 > > > > org.apache.drill.exec.physical.impl.BaseRootExec.next():104 > > > > > > > > > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93 > > > > org.apache.drill.exec.physical.impl.BaseRootExec.next():94 > > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():256 > > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():250 > > > > java.security.AccessController.doPrivileged():-2 > > > > javax.security.auth.Subject.doAs():415 > > > > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > > > > org.apache.drill.exec.work.fragment.FragmentExecutor.run():250 > > > > org.apache.drill.common.SelfCleaningRunnable.run():38 > > > > java.util.concurrent.ThreadPoolExecutor.runWorker():1145 > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run():615 > > > > java.lang.Thread.run():745 (state=,code=0) > > > -- Sent from my iThing
