[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319271#comment-16319271
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1082#discussion_r160539584
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -180,18 +251,46 @@ else
fi
fi
-# Default memory settings if none provided by the environment or
+# Checking if being executed in context of Drillbit and not SQLLine
+if [ "$DRILLBIT_CONTEXT" == "1" ]; then
+ # *-auto.sh allows for distrib/user specific checks to be done
+ distribAuto="$DRILL_CONF_DIR/distrib-auto.sh"
+ if [ ! -r "$distribAuto" ]; then
distribAuto="$DRILL_HOME/conf/distrib-auto.sh"; fi
+ if [ ! -r "$distribAuto" ]; then distribAuto=""; fi
+ drillAuto="$DRILL_CONF_DIR/drill-auto.sh"
+ if [ ! -r "$drillAuto" ]; then
drillAuto="$DRILL_HOME/conf/drill-auto.sh"; fi
--- End diff --
This part is just a bit tricky. Drill- and distribution- specific files do,
in fact, reside in `$DRILL_HOME/conf`. But, user files reside in a number of
places. So, please use:
- `$DRILL_HOME/conf` for `distrib-auto.sh`
- `$DRILL_CONF_DIR` for user-supplied files (such as `drill-auto.sh`)
> Automatically manage memory allocations during startup
> ------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.13.0
>
> Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf,
> Auto Mem Allocation Proposal - Scenarios.pdf
>
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on
> configuration patterns observed in production Drill clusters). It leverages
> the capability of providing distribution (and user-specific) checks during
> Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning
> parameters, by providing the optimal values. In addition, it also allows for
> the memory allocation to be implicitly managed by simply providing the Drill
> process with a single dimensional of total process memory (either in absolute
> values, or as a percentage of the total system memory), while
> {{distrib-auto.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct
> Memory, with a small portion allocated for the Generated Java CodeCache as
> well. If any of the individual allocations are also specified (via
> {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified
> allocations are adjusted to stay +within the limits+ of the total memory
> allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the
> equation used for computing the heap, direct and code cache allocations for a
> given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various
> inputs, and their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}}
> or {{drill-env.sh}} ) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account
> for this any more, and will unset the variable if JDK8 or higher is detected._
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
