Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/938#discussion_r138437442
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
---
@@ -646,6 +687,46 @@ public AggOutcome doWork() {
}
/**
+ * Use reserved values memory (if available) to try and preemp an OOM
+ */
+ private void useReservedValuesMemory() {
+ // try to preempt an OOM by using the reserved memory
+ long reservedMemory = reserveValueBatchMemory;
+ if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() +
reservedMemory); }
+
+ reserveValueBatchMemory = 0;
+ }
+ /**
+ * Use reserved outgoing output memory (if available) to try and
preemp an OOM
+ */
+ private void useReservedOutgoingMemory() {
+ // try to preempt an OOM by using the reserved memory
+ long reservedMemory = reserveOutgoingMemory;
+ if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() +
reservedMemory); }
--- End diff --
Because the first uncontrolled memory allocation happens when inserting
into the hash table (i.e. put()). Given this "uncontrollability", better OOM
there (which we can handle, by spilling and retrying). Now if all the memory
was "given" in the limit, the put() may not OOM, but leave too little available
memory to continue (i.e. to create a values batch, or an outgoing batch) --
these situations we can not handle.
By subtracting from the limit a "reserve" for these two batches, we may
force a put() OOM early (but that's OK). But we also ensure that the following
two batches could be allocated. In some way this is similar to having multiple
dedicated allocators, only simpler.
Adding or subtracting is just an operation on a local field; no
performance effect in any way. Also by using a single allocator we can handle
cases like a "bump" in a batch size (which may exceed pre-allocation in a
dedicated separate allocator).
---
