[ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319829#comment-16319829 ]
ASF GitHub Bot commented on DRILL-5741: --------------------------------------- Github user kkhatua commented on a diff in the pull request: https://github.com/apache/drill/pull/1082#discussion_r160606258 --- Diff: distribution/src/resources/distrib-auto.sh --- @@ -0,0 +1,223 @@ +#!/usr/bin/env bash --- End diff -- Wasn't sure which would be a a suitable place. I reasoned the "Apache" distribution is what we are defining it for, hence the choice > Automatically manage memory allocations during startup > ------------------------------------------------------ > > Key: DRILL-5741 > URL: https://issues.apache.org/jira/browse/DRILL-5741 > Project: Apache Drill > Issue Type: Improvement > Components: Server > Affects Versions: 1.11.0 > Reporter: Kunal Khatua > Assignee: Kunal Khatua > Fix For: 1.13.0 > > Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf, > Auto Mem Allocation Proposal - Scenarios.pdf > > > Currently, during startup, a Drillbit can be assigned large values for the > following: > * Xmx (Heap) > * XX:MaxDirectMemorySize > * XX:ReservedCodeCacheSize > * XX:MaxPermSize > All of this, potentially, can exceed the available memory on a system when a > Drillbit is under heavy load. It would be good to have the Drillbit ensure > during startup itself that the cumulative value of these parameters does not > exceed a pre-defined upper limit for the Drill process. > This JIRA is a *proposal* to allow for automatic configuration (based on > configuration patterns observed in production Drill clusters). It leverages > the capability of providing distribution (and user-specific) checks during > Drill Startup from DRILL-6068. > The idea is to remove the need for a user to worry about managing the tuning > parameters, by providing the optimal values. In addition, it also allows for > the memory allocation to be implicitly managed by simply providing the Drill > process with a single dimensional of total process memory (either in absolute > values, or as a percentage of the total system memory), while > {{distrib-auto.sh}} provides the individual allocations. > This allocation is then partitioned into allocations for Heap and Direct > Memory, with a small portion allocated for the Generated Java CodeCache as > well. If any of the individual allocations are also specified (via > {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified > allocations are adjusted to stay +within the limits+ of the total memory > allocation. > The *details* of the proposal are here: > https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0 > For those unable to access the Google Document, PDFs are attached: > * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the > equation used for computing the heap, direct and code cache allocations for a > given input > * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various > inputs, and their expected allocations > The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}} > or {{drill-env.sh}} ) are: > * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory > * {{DRILL_HEAP}} : JVM Max Heap Size > * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size > * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size > Note: _With JDK8, MaxPermSize is no longer supported, so we do not account > for this any more, and will unset the variable if JDK8 or higher is detected._ -- This message was sent by Atlassian JIRA (v6.4.14#64029)