[
https://issues.apache.org/jira/browse/SPARK-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351503#comment-14351503
]
Apache Spark commented on SPARK-6214:
-------------------------------------
User 'tigerquoll' has created a pull request for this issue:
https://github.com/apache/spark/pull/4937
> Allow configuration options to use a simple expression language
> ---------------------------------------------------------------
>
> Key: SPARK-6214
> URL: https://issues.apache.org/jira/browse/SPARK-6214
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Dale Richardson
> Priority: Minor
>
> This is a proposal to allow for configuration options to be specified via a
> simple expression language. This language would have the following features:
> * Allow for basic arithmetic (+-/*) with support bracketed expressions and
> standard precedence rules.
> * Support for and normalisation of common units of reference eg. MB, GB,
> seconds,minutes,hours, days and weeks.
> * Allow for the referencing of basic environmental information currently
> defined as:
> numCores: Number of cores assigned to the JVM
> physicalMemoryBytes: Memory size of hosting machine
> JVMTotalMemoryBytes: current bytes of memory allocated to the JVM
> JVMMaxMemoryBytes: Maximum number of bytes of memory available to the JVM
> JVMFreeMemoryBytes: maxMemoryBytes - totalMemoryBytes
> * Allow for the limited referencing of other configuration values when
> specifying values. (Other configuration values must be initialised and
> explicitly passed into the expression evaluator for this functionality to be
> enabled).
>
> Such a feature would have the following end-user benefits:
> * Allow for the flexibility in specifying time intervals or byte quantities
> in appropriate and easy to follow units e.g. 1 week rather rather then
> 604800 seconds
> * Have a consistent means of entering configuration information regardless of
> the configuration option being added. (eg questions such as ‘is the
> particular option specified in ms or seconds?’ become irrelevant, because the
> user can pick what ever unit makes sense for the magnitude of the value they
> are specifying)
> * Allow for the scaling of a configuration option in relation to a system
> attributes. e.g.
> SPARK_WORKER_CORES = numCores - 1
> SPARK_WORKER_MEMORY = physicalMemoryBytes - 1.5 GB
> * Being able to scale multiple configuration options together eg:
> spark.driver.memory = 0.75 * physicalMemoryBytes
> spark.driver.maxResultSize = spark.driver.memory * 0.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
