Hanifi,
I just bumped into this as well.
Error: SYSTEM ERROR: OversizedAllocationException: Unable to expand the
buffer. Max allowed buffer size is reached.
Earlier in the thread you say:
> You see this exception because one of the columns in your dataset is larger
> than an individual DrillBuf could store. The hard limit
> is Integer.MAX_VALUE bytes.
So is it right to say the maximum buffer size is 2GB? I'm getting this
exception on a data set whose *total* size is less than 1GB - as reported
by "du -sh" on the top level directory I am querying. So I'm confused.
I have a guess as to which column in my dataset is causing the problem.
It's likely a substantial JSON document that comes from a file, and the
size of that file varies widely. I process the file into a dictionary in
Python before writing it to my workspace in a format that works but for
this issue. The largest of these documents weighs in at only 320KB.
I could go down the path of reshaping the large document so that Drill sees
multiple columns, but I don't see how I can be sure that will work since
all of the columns in my data are so far below Integer.MAX_VALUE bytes.
Is there any other recommendation you can make, apart from further ETLing
the data?
--vince
----
Vince Gonzalez
Systems Engineer
212.694.3879
mapr.com
On Mon, Feb 8, 2016 at 2:05 PM, Hanifi Gunes <hgu...@maprtech.com> wrote:
> Thanks for the feedback. Yep my answer seems very much dev focused than
> user.
>
> The error is manifestation of extremely wide columns in your dataset. I
> would recommend splitting the list if that's an option.
>
> Assuming the problem column is a list of integers as below
>
> {
> "wide": [1,2,.....N]
> }
>
> after splitting it should look like
>
> {
> "wide0": [1,2,.....X],
> "wide1": [Y,.......Z]
> ...
> "wideN": [T,.......N]
> }
>
> Sounds like good idea to enhance the error reporting with file & column
> name. Filed [1] to track this.
>
> Thanks.
>
> 1: https://issues.apache.org/jira/browse/DRILL-4371
>
>
> On Fri, Feb 5, 2016 at 6:28 PM, John Omernik <j...@omernik.com> wrote:
>
> > Excuse my basic questions, when you say we are you reference Drill
> coders?
> > So what is Integer.MAX_VALUE bytes? Is that a query time setting?
> Drillbit
> > setting? Is it editable? How does that value get interpreted for complex
> > data types (objects and arrays).
> >
> > Not only would the column be helpful, but the source file as well. ( if
> > this is an individual record issue....or is this a cumulative error where
> > the max size of the sum of the length of multiple records of a column is
> at
> > issue).
> >
> >
> > Thoughts on how as a user I could address this in my dataset?
> >
> > Thanks!
> >
> > On Friday, February 5, 2016, Hanifi Gunes <hgu...@maprtech.com> wrote:
> >
> > > You see this exception because one of the columns in your dataset is
> > larger
> > > than an individual DrillBuf could store. The hard limit
> > > is Integer.MAX_VALUE bytes. Around the time we are trying to expand one
> > of
> > > the buffers, we notice the allocation request is oversized and fail the
> > > query. It would be nice if error message contained the column that
> raised
> > > this issue though.
> > >
> > > On Fri, Feb 5, 2016 at 1:39 PM, John Omernik <j...@omernik.com
> > > <javascript:;>> wrote:
> > >
> > > > Any thoughts on how to troubleshoot this (I have some fat json data
> > going
> > > > into the buffers apparently) It's not huge data, just wide/complex
> > (total
> > > > size is 1.4 GB) Any thoughts on how to troubleshoot or settings I
> can
> > > use
> > > > to work through these errors?
> > > >
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > Error: SYSTEM ERROR: OversizedAllocationException: Unable to expand
> the
> > > > buffer. Max allowed buffer size is reached.
> > > >
> > > >
> > > >
> > > > Fragment 1:11
> > > >
> > > >
> > > >
> > > > [Error Id: db21dea0-ddd7-4fcf-9fea-b5031e358dad on node1
> > > >
> > > >
> > > >
> > > > (org.apache.drill.exec.exception.OversizedAllocationException)
> Unable
> > > to
> > > > expand the buffer. Max allowed buffer size is reached.
> > > >
> > > > org.apache.drill.exec.vector.UInt1Vector.reAlloc():214
> > > >
> > > >
> > org.apache.drill.exec.vector.UInt1Vector$Mutator.setValueCount():469
> > > >
> > > >
> > > >
> > >
> >
> org.apache.drill.exec.vector.complex.ListVector$Mutator.setValueCount():324
> > > >
> > > > org.apache.drill.exec.physical.impl.ScanBatch.next():247
> > > >
> > > > org.apache.drill.exec.record.AbstractRecordBatch.next():119
> > > >
> > > > org.apache.drill.exec.record.AbstractRecordBatch.next():109
> > > >
> > > >
> > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> > > >
> > > >
> > > >
> > > >
> > >
> >
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132
> > > >
> > > > org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > > >
> > > > org.apache.drill.exec.record.AbstractRecordBatch.next():119
> > > >
> > > >
> > > >
> > > >
> > >
> >
> org.apache.drill.exec.test.generated.StreamingAggregatorGen1931.doWork():172
> > > >
> > > >
> > > >
> > > >
> > >
> >
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():167
> > > >
> > > > org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > > >
> > > > org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> > > >
> > > >
> > > >
> > > >
> > >
> >
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
> > > >
> > > > org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> > > >
> > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():256
> > > >
> > > > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():250
> > > >
> > > > java.security.AccessController.doPrivileged():-2
> > > >
> > > > javax.security.auth.Subject.doAs():415
> > > >
> > > > org.apache.hadoop.security.UserGroupInformation.doAs():1595
> > > >
> > > > org.apache.drill.exec.work.fragment.FragmentExecutor.run():250
> > > >
> > > > org.apache.drill.common.SelfCleaningRunnable.run():38
> > > >
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> > > >
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> > > >
> > > > java.lang.Thread.run():745 (state=,code=0)
> > > >
> > >
> >
> >
> > --
> > Sent from my iThing
> >
>
