[
https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342841#comment-16342841
]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1082#discussion_r164328389
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -180,18 +251,61 @@ else
fi
fi
-# Default memory settings if none provided by the environment or
+# Execute distrib-setup.sh for any distribution-specific setup (e.g.
checks).
+# distrib-setup.sh is optional; it is created by some distribution
installers
+# that need additional distribution-specific setup to be done.
+# Because installers will have site-specific steps, the file
+# should be moved into the site directory, if the user employs one.
+
+# Checking if being executed in context of Drillbit and not SQLLine
+if [ "$DRILLBIT_CONTEXT" == "1" ]; then
+ # Check whether to run exclusively distrib-setup.sh OR auto-setup.sh
+ distribSetup="$DRILL_CONF_DIR/distrib-setup.sh" ; #Site-based
distrib-setup.sh
+ if [ $(checkExecutableLineCount $distribSetup) -eq 0 ]; then
+ distribSetup="$DRILL_HOME/conf/distrib-setup.sh" ; #Install-based
distrib-setup.sh
+ if [ $(checkExecutableLineCount $distribSetup) -eq 0 ]; then
+ # Run Default Auto Setup
+ distribSetup="$DRILL_HOME/bin/auto-setup.sh"
+ fi
+ fi
+ # Check and run additional setup defined by user
+ drillSetup="$DRILL_CONF_DIR/drill-setup.sh" ; #Site-based drill-setup.sh
+ if [ $(checkExecutableLineCount $drillSetup) -eq 0 ]; then
+ drillSetup="$DRILL_HOME/conf/drill-setup.sh" ; #Install-based
drill-setup.sh
+ if [ $(checkExecutableLineCount $drillSetup) -eq 0 ]; then
drillSetup=""; fi
+ fi
+
+ # Enforcing checks in order (distrib-setup.sh , drill-setup.sh)
+ # (NOTE: A script is executed only if it has relevant executable lines)
+ # Both distribSetup & drillSetup are executed because the user might
have introduced additional checks
+ if [ -n "$distribSetup" ]; then
+ . "$distribSetup"
+ if [ $? -gt 0 ]; then fatal_error "Aborting Drill Startup due failed
setup by $distribSetup"; fi
--- End diff --
In the interests of simplicity, I would just have the script itself report
errors and exit. See the existing code in `drill-config.sh`. This way, the
script can give a reasonable error message. And, it is not very hard to write
`exit 1` to exit the scripts.
> Automatically manage memory allocations during startup
> ------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.13.0
>
> Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf,
> Auto Mem Allocation Proposal - Scenarios.pdf
>
>
> Currently, during startup, a Drillbit can be assigned large values for the
> following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a
> Drillbit is under heavy load. It would be good to have the Drillbit ensure
> during startup itself that the cumulative value of these parameters does not
> exceed a pre-defined upper limit for the Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on
> configuration patterns observed in production Drill clusters). It leverages
> the capability of providing default/distribution (and user-specific) checks
> during Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning
> parameters, by providing the optimal values. In addition, it also allows for
> the memory allocation to be implicitly managed by simply providing the Drill
> process with a single dimensional of total process memory (either in absolute
> values, or as a percentage of the total system memory), while
> {{auto-setup.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct
> Memory, with a small portion allocated for the Generated Java CodeCache as
> well. If any of the individual allocations are also specified (via
> {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified
> allocations are adjusted to stay +within the limits+ of the total memory
> allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the
> equation used for computing the heap, direct and code cache allocations for a
> given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various
> inputs, and their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}}
> or {{drill-env.sh}} ) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account
> for this any more, and will unset the variable if JDK8 or higher is detected._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
