[ https://issues.apache.org/jira/browse/DRILL-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180067#comment-16180067 ]
ASF GitHub Bot commented on DRILL-5808: --------------------------------------- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/958#discussion_r140939339 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java --- @@ -22,16 +22,55 @@ import javax.annotation.concurrent.ThreadSafe; import org.apache.drill.exec.exception.OutOfMemoryException; +import org.apache.drill.exec.util.AssertionUtil; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; /** * Provides a concurrent way to manage account for memory usage without locking. Used as basis for Allocators. All * operations are threadsafe (except for close). */ @ThreadSafe -class Accountant implements AutoCloseable { - // private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class); +@VisibleForTesting +public class Accountant implements AutoCloseable { + private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class); + + // See DRILL-5808 + // Allow a "grace margin" above the allocator limit for those operators + // that are trying to stay within the limit, but are hindered by the + // current power-of-two allocations and random vector doublings. Instead + // of failing the query, the accountant allows an excess allocation within + // the grace, but issues a warning to the log. + // + // Grace is allowed only when requested, and then only in production + // code, or if enabled in debug code by setting the following system + // property. + + public static final String STRICT_ALLOCATOR = "drill.memory.strict.allocator"; --- End diff -- The tricky-bit is that the allocator/accountant is below the level of java-exec, and so does not have visibility to either Drill config or the option manager. So, this has to be set via a system option, as in the other memory manager controls. A possible enhancement would be to grab the options in the Drill server startup (or main) and set statics down in the memory layer. > Reduce memory allocator strictness for "managed" operators > ---------------------------------------------------------- > > Key: DRILL-5808 > URL: https://issues.apache.org/jira/browse/DRILL-5808 > Project: Apache Drill > Issue Type: Improvement > Affects Versions: 1.11.0 > Reporter: Paul Rogers > Assignee: Paul Rogers > Fix For: 1.12.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Drill 1.11 and 1.12 introduce new "managed" versions of the sort and hash agg > that enforce memory limits, spilling to disk when necessary. > Drill's internal memory system is very "lumpy" and unpredictable. The > operators have no control over the incoming batch size; an overly large batch > can cause the operator to exceed its memory limit before it has a chance to > do any work. > Vector allocations grow in power-of-two sizes. Adding a single record can > double the memory allocated to a vector. > Drill has no metadata, so operators cannot predict the size of VarChar > columns nor the cardinality of arrays. The "Record Batch Sizer" tries to > extract this information on each batch, but it works with averages, and > specific column patterns can still throw off the memory calculations. (For > example, having a series of very wide columns for A-M and very narrow columns > for N-Z will cause a moderate average. But, once sorted, the A-M rows, and > batches, will be much larger than expected, causing out-of-memory errors.) > At present, if an operator is wrong in its memory usage by a single byte, the > entire query is killed. That is, the user pays the death penalty (of queries) > for poor design decisions within Drill. This leads to a less-than-optimal > user experience. > The proposal here is to make the memory allocator less strict for "managed" > operators. > First, we recognize that the managed operators do attempt to control memory > and, if designed well, will, on average hit their targets. > Second, we recognize that, due to the lumpiness issues above, any single > operator may exceed, or be under, the configured maximum memory. > Given this, the proposal here is: > 1. An operator identifies itself as managed to the memory allocator. > 2. In managed mode, the allocator has soft limits. It emits a warning to the > log when the limit is exceeded. > 3. For safety, in managed mode, the allocator enforces a hard limit larger > than the configured limit. > The enforcement limit might be: > * For memory sizes < 100MB, up to 2x the configured limit. > * For larger memory sizes, no more than 100MB over the configured limit. > The exact numbers can be made configurable. > Now, during testing, scripts should look for over-memory warnings. Each > should be fixed as we fix OOM issues today. But, during production, user > queries are far less likely to fail due to any remaining corner cases that > throw off the memory calculations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)