[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334890#comment-15334890 ] Kunal Khatua commented on DRILL-4236: - Setting direct memory to 10GB and altered the session by setting `planner.memory.max_query_memory_per_node`=21474836 (20MB). The data begins to spill to disk with no significant increase in the Drillbit memory footprint. Verified on Drill 1.6.0. > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: documentation > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098449#comment-15098449 ] Jacques Nadeau commented on DRILL-4236: --- This is a separate bug we should file. The external sort operation needs to reserve any necessary memory and ensure that it doesn't ever hit an out of memory condition. [~adeneche] & [~sphillips], it seems like we need to go through with a fine tooth comb and ensure that all possible memory allocations within the External sort are correctly handled. It may make sense to break into two possible situations: easily accommodated and hard to accommodate. For the latter, we probably need to create a separate child allocator with a reservation and usage policy that ensures we don't hit out of memory. > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: documentation > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098498#comment-15098498 ] Victoria Markman commented on DRILL-4236: - We have an enhancement request for sort with merge joins: https://issues.apache.org/jira/browse/DRILL-4181 (I believe that this is the case I'm running into) > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: documentation > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098466#comment-15098466 ] Victoria Markman commented on DRILL-4236: - I agree that it's a separate bug and I was combing jira to find existing bug, because it's always been the case that sort could run out of memory in some specific conditions. Is this something we are going to fix for 1.5.0 and hold release for ? > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: documentation > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098518#comment-15098518 ] Jacques Nadeau commented on DRILL-4236: --- I guess it depends on how much more frequent the problems are. Do you have a sense of this? I think this is structural and has existed for some time so my first inclination is that it is not a release blocker. That said, if, even with the config changes, this is substantially more frequent, we should probably take stock. > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: documentation > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[
https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098434#comment-15098434
]
Victoria Markman commented on DRILL-4236:
-
This fix might cause query failures after upgrade to 1.5.0 due to different
accounting for memory allocated by sort operator.
I believe this either:
* Needs to be documented and stated that workaround is either increasing
planner.memory.max_query_memory_per_node or reducing planner.width.max_per_node
(which is not a very desirable option in my opinion)
or
* Fixing https://issues.apache.org/jira/browse/DRILL-3549 by calculating
memory allocated to sort operator as some sort of formula of percentage of
direct memory and planner.width.max_per_node
For the reference, here is the error I'm running into:
{code}
2016-01-14 16:38:26,389 [2968310e-3c6d-3bb5-0a3c-90fbc178aa2f:frag:12:0] INFO
o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: External Sort
encountered an error while spilling to disk
[Error Id: 2c517e51-f097-4e4d-ae93-7f1839cc813a ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:569)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:389)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordIterator.nextBatch(RecordIterator.java:97)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordIterator.next(RecordIterator.java:183)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordIterator.prepare(RecordIterator.java:167)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.JoinStatus.prepare(JoinStatus.java:86)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:162)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at
[jira] [Commented] (DRILL-4236) ExternalSort should use the new allocator functionality to better manage it's memory usage
[ https://issues.apache.org/jira/browse/DRILL-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083711#comment-15083711 ] Deneche A. Hakim commented on DRILL-4236: - created PR [#317|https://github.com/apache/drill/pull/317] [~sphillips] can you please review ? > ExternalSort should use the new allocator functionality to better manage it's > memory usage > -- > > Key: DRILL-4236 > URL: https://issues.apache.org/jira/browse/DRILL-4236 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.5.0 >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Fix For: 1.5.0 > > > Before DRILL-4215, the sort operator wasn't able to correctly compute it's > memory usage, and so it jumped through a bunch of hoops to try to figure out > when it should spill to disk. > With the transfer accounting in place, this code can be greatly simplified to > just use the current operator memory allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
