Chun Chang created DRILL-5694:
---------------------------------
Summary: hash agg spill to disk, second phase OOM
Key: DRILL-5694
URL: https://issues.apache.org/jira/browse/DRILL-5694
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.11.0
Reporter: Chun Chang
Assignee: Boaz Ben-Zvi
| 1.11.0-SNAPSHOT | d622f76ee6336d97c9189fc589befa7b0f4189d6 | DRILL-5165:
For limit all case, no need to push down limit to scan | 21.07.2017 @ 10:36:29
PDT
Second phase agg ran out of memory. Not suppose to. Test data currently only
accessible locally.
/root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
Query:
select row_count, sum(row_count), avg(double_field), max(double_rand),
count(float_rand) from parquet_500m_v1 group by row_count order by row_count
limit 30
Failed with exception
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
while executing the query.
HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size:
4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit:
536870912 so far allocated: 534773760.
Fragment 1:6
[Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
(org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 OOM
at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned
batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far
allocated: 534773760.
org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to
allocate buffer of size 4194304 due to memory limit. Current allocation:
534773760
org.apache.drill.exec.memory.BaseAllocator.buffer():238
org.apache.drill.exec.memory.BaseAllocator.buffer():213
org.apache.drill.exec.vector.IntVector.allocateBytes():231
org.apache.drill.exec.vector.IntVector.allocateNew():211
org.apache.drill.exec.test.generated.HashTableGen2141.allocMetadataVector():778
org.apache.drill.exec.test.generated.HashTableGen2141.resizeAndRehashIfNeeded():717
org.apache.drill.exec.test.generated.HashTableGen2141.insertEntry():643
org.apache.drill.exec.test.generated.HashTableGen2141.put():618
org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1173
org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:220)
at
org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:101)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException:
RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size:
4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit:
536870912 so far allocated: 534773760.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
