This is clearly a bug and like zelaine suggested the new sort is still work
in progress. We have a few similar bugs open for the new sort. I could have
pointed to the jira's but unfortunately JIRA is not working for me due to
firewall issues.
Another suggestion is build drill from the latest master and try it out, if
you are willing to spend some time. But again there is no guarantee yet.
Please go ahead and raise a new jira. If it is a duplicate, I will mark it
as such later. Thank You.
- Rahul
On Tue, May 2, 2017 at 8:24 AM, Nate Butler <n...@chartbeat.com> wrote:
> Zelaine, thanks for the suggestion. I added this option both to the
> drill-override and in the session and this time the query did stay running
> for much longer but it still eventually failed with the same error,
> although much different memory values.
>
> (org.apache.drill.exec.exception.OutOfMemoryException) Unable to
> allocate
> buffer of size 134217728 due to memory limit. Current allocation:
> 10653214316
> org.apache.drill.exec.memory.BaseAllocator.buffer():220
> org.apache.drill.exec.memory.BaseAllocator.buffer():195
> org.apache.drill.exec.vector.VarCharVector.reAlloc():425
> org.apache.drill.exec.vector.VarCharVector.copyFromSafe():278
> org.apache.drill.exec.vector.NullableVarCharVector.copyFromSafe():379
> org.apache.drill.exec.test.generated.PriorityQueueCopierGen8.
> doCopy():22
> org.apache.drill.exec.test.generated.PriorityQueueCopierGen8.next():76
>
> org.apache.drill.exec.physical.impl.xsort.managed.
> CopierHolder$BatchMerger.next():234
>
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.
> doMergeAndSpill():1408
>
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.
> mergeAndSpill():1376
>
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.
> spillFromMemory():1339
>
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.
> processBatch():831
>
> org.apache.drill.exec.physical.impl.xsort.managed.
> ExternalSortBatch.loadBatch():618
>
> org.apache.drill.exec.physical.impl.xsort.managed.
> ExternalSortBatch.load():660
>
> org.apache.drill.exec.physical.impl.xsort.managed.
> ExternalSortBatch.innerNext():559
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
> org.apache.drill.exec.physical.impl.aggregate.
> StreamingAggBatch.innerNext():137
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
>
> org.apache.drill.exec.physical.impl.partitionsender.
> PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1657
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
>
> At first I didn't change planner.width.max_per_query and the default on a
> 32 core machine makes it 23. This query failed after 34 minutes. I then
> tried setting planner.width.max_per_query=1 and this query also failed but
> of course took took longer, about 2 hours. In both cases,
> planner.memory.max_query_memory_per_node was set to 230G.
>
>
> On Mon, May 1, 2017 at 11:09 AM, Zelaine Fong <zf...@mapr.com> wrote:
>
> > Nate,
> >
> > The Jira you’ve referenced relates to the new external sort, which is not
> > enabled by default, as it is still going through some additional testing.
> > If you’d like to try it to see if it resolves your problem, you’ll need
> to
> > set “sort.external.disable_managed” as follows in your
> > drill-override.conf file:
> >
> > drill.exec: {
> > cluster-id: "drillbits1",
> > zk.connect: "localhost:2181",
> > sort.external.disable_managed: false
> > }
> >
> > and run the following query:
> >
> > ALTER SESSION SET `exec.sort.disable_managed` = false;
> >
> > -- Zelaine
> >
> > On 5/1/17, 7:44 AM, "Nate Butler" <n...@chartbeat.com> wrote:
> >
> > We keep running into this issue when trying to issue a query with
> > hashagg
> > disabled. When I look at system memory usage though, drill doesn't
> > seem to
> > be using much of it but still hits this error.
> >
> > Our environment:
> >
> > - 1 r3.8xl
> > - 1 drillbit version 1.10.0 configured with 4GB of Heap and 230G of
> > Direct
> > - Data stored on S3 is compressed CSV
> >
> > I've tried increasing planner.memory.max_query_memory_per_node to
> > 230G and
> > lowered planner.width.max_per_query to 1 and it still fails.
> >
> > We've applied the patch from this bug in the hopes that it would
> > resolve
> > the issue but it hasn't:
> >
> > https://issues.apache.org/jira/browse/DRILL-5226
> >
> > Stack Trace:
> >
> > (org.apache.drill.exec.exception.OutOfMemoryException) Unable to
> > allocate
> > buffer of size 16777216 due to memory limit. Current allocation:
> > 8445952
> > org.apache.drill.exec.memory.BaseAllocator.buffer():220
> > org.apache.drill.exec.memory.BaseAllocator.buffer():195
> > org.apache.drill.exec.vector.VarCharVector.reAlloc():425
> > org.apache.drill.exec.vector.VarCharVector.copyFromSafe():278
> > org.apache.drill.exec.vector.NullableVarCharVector.
> > copyFromSafe():379
> >
> > org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.
> > doCopy():22
> > org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.
> > next():75
> >
> > org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.
> > mergeAndSpill():602
> >
> > org.apache.drill.exec.physical.impl.xsort.
> > ExternalSortBatch.innerNext():428
> > org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > org.apache.drill.exec.record.AbstractRecordBatch.next():119
> > org.apache.drill.exec.record.AbstractRecordBatch.next():109
> >
> > org.apache.drill.exec.physical.impl.aggregate.
> > StreamingAggBatch.innerNext():137
> > org.apache.drill.exec.record.AbstractRecordBatch.next():162
> > org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> >
> > org.apache.drill.exec.physical.impl.partitionsender.
> > PartitionSenderRootExec.innerNext():144
> > org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
> > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
> > java.security.AccessController.doPrivileged():-2
> > javax.security.auth.Subject.doAs():422
> > org.apache.hadoop.security.UserGroupInformation.doAs():1657
> > org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
> > org.apache.drill.common.SelfCleaningRunnable.run():38
> > java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> > java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> > java.lang.Thread.run():745 (state=,code=0)
> >
> > Is there something I'm missing here? Any help/direction would be
> > appreciated.
> >
> > Thanks,
> > Nate
> >
> >
> >
>
