[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353161#comment-16353161
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
Github user kkhatua commented on a diff in the pull request:
https://github.com/apache/drill/pull/1082#discussion_r166157864
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -180,18 +251,61 @@ else
fi
fi
-# Default memory settings if none provided by the environment or
+# Execute distrib-setup.sh for any distribution-specific setup (e.g.
checks).
+# distrib-setup.sh is optional; it is created by some distribution
installers
+# that need additional distribution-specific setup to be done.
+# Because installers will have site-specific steps, the file
+# should be moved into the site directory, if the user employs one.
+
+# Checking if being executed in context of Drillbit and not SQLLine
+if [ "$DRILLBIT_CONTEXT" == "1" ]; then
+ # Check whether to run exclusively distrib-setup.sh OR auto-setup.sh
+ distribSetup="$DRILL_CONF_DIR/distrib-setup.sh" ; #Site-based
distrib-setup.sh
+ if [ $(checkExecutableLineCount $distribSetup) -eq 0 ]; then
--- End diff --
I'd have liked the KISS principle, but I thought there was a need for
placeholder `distrib-setup.sh` file.
Based on that, I need to figure out whether there is a distribtion-specific
setup, or should we revert to executing the `auto-setup.sh`. Unlike sourcing
environment files, where an unset variable can be set, for auto-setup, the
choice of execution has to be mutually exclusive.
This block looks complicated (and verbose with the comments), but is only
identifying *what* setup script needs to execute. Hence, all we do here is an
assignment of the variables.
> Automatically manage memory allocations during startup
> ------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.13.0
>
> Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf,
> Auto Mem Allocation Proposal - Scenarios.pdf
>
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on
> configuration patterns observed in production Drill clusters). It leverages
> the capability of providing default/distribution (and user-specific) checks
> during Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning
> parameters, by providing the optimal values. In addition, it also allows for
> the memory allocation to be implicitly managed by simply providing the Drill
> process with a single dimensional of total process memory (either in absolute
> values, or as a percentage of the total system memory), while
> {{auto-setup.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct
> Memory, with a small portion allocated for the Generated Java CodeCache as
> well. If any of the individual allocations are also specified (via
> {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified
> allocations are adjusted to stay +within the limits+ of the total memory
> allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the
> equation used for computing the heap, direct and code cache allocations for a
> given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various
> inputs, and their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}}
> or {{drill-env.sh}} ) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account
> for this any more, and will unset the variable if JDK8 or higher is detected._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
