[
https://issues.apache.org/jira/browse/DRILL-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180067#comment-16180067
]
ASF GitHub Bot commented on DRILL-5808:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/958#discussion_r140939339
--- Diff:
exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java ---
@@ -22,16 +22,55 @@
import javax.annotation.concurrent.ThreadSafe;
import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.util.AssertionUtil;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
/**
* Provides a concurrent way to manage account for memory usage without
locking. Used as basis for Allocators. All
* operations are threadsafe (except for close).
*/
@ThreadSafe
-class Accountant implements AutoCloseable {
- // private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+@VisibleForTesting
+public class Accountant implements AutoCloseable {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+
+ // See DRILL-5808
+ // Allow a "grace margin" above the allocator limit for those operators
+ // that are trying to stay within the limit, but are hindered by the
+ // current power-of-two allocations and random vector doublings. Instead
+ // of failing the query, the accountant allows an excess allocation
within
+ // the grace, but issues a warning to the log.
+ //
+ // Grace is allowed only when requested, and then only in production
+ // code, or if enabled in debug code by setting the following system
+ // property.
+
+ public static final String STRICT_ALLOCATOR =
"drill.memory.strict.allocator";
--- End diff --
The tricky-bit is that the allocator/accountant is below the level of
java-exec, and so does not have visibility to either Drill config or the option
manager. So, this has to be set via a system option, as in the other memory
manager controls.
A possible enhancement would be to grab the options in the Drill server
startup (or main) and set statics down in the memory layer.
> Reduce memory allocator strictness for "managed" operators
> ----------------------------------------------------------
>
> Key: DRILL-5808
> URL: https://issues.apache.org/jira/browse/DRILL-5808
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Drill 1.11 and 1.12 introduce new "managed" versions of the sort and hash agg
> that enforce memory limits, spilling to disk when necessary.
> Drill's internal memory system is very "lumpy" and unpredictable. The
> operators have no control over the incoming batch size; an overly large batch
> can cause the operator to exceed its memory limit before it has a chance to
> do any work.
> Vector allocations grow in power-of-two sizes. Adding a single record can
> double the memory allocated to a vector.
> Drill has no metadata, so operators cannot predict the size of VarChar
> columns nor the cardinality of arrays. The "Record Batch Sizer" tries to
> extract this information on each batch, but it works with averages, and
> specific column patterns can still throw off the memory calculations. (For
> example, having a series of very wide columns for A-M and very narrow columns
> for N-Z will cause a moderate average. But, once sorted, the A-M rows, and
> batches, will be much larger than expected, causing out-of-memory errors.)
> At present, if an operator is wrong in its memory usage by a single byte, the
> entire query is killed. That is, the user pays the death penalty (of queries)
> for poor design decisions within Drill. This leads to a less-than-optimal
> user experience.
> The proposal here is to make the memory allocator less strict for "managed"
> operators.
> First, we recognize that the managed operators do attempt to control memory
> and, if designed well, will, on average hit their targets.
> Second, we recognize that, due to the lumpiness issues above, any single
> operator may exceed, or be under, the configured maximum memory.
> Given this, the proposal here is:
> 1. An operator identifies itself as managed to the memory allocator.
> 2. In managed mode, the allocator has soft limits. It emits a warning to the
> log when the limit is exceeded.
> 3. For safety, in managed mode, the allocator enforces a hard limit larger
> than the configured limit.
> The enforcement limit might be:
> * For memory sizes < 100MB, up to 2x the configured limit.
> * For larger memory sizes, no more than 100MB over the configured limit.
> The exact numbers can be made configurable.
> Now, during testing, scripts should look for over-memory warnings. Each
> should be fixed as we fix OOM issues today. But, during production, user
> queries are far less likely to fail due to any remaining corner cases that
> throw off the memory calculations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
