Marcelo Vanzin created SPARK-16272:
--------------------------------------
Summary: Allow configs to reference other configs, env and system
properties
Key: SPARK-16272
URL: https://issues.apache.org/jira/browse/SPARK-16272
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Marcelo Vanzin
Priority: Minor
Currently, Spark's configuration is static; it is whatever is written to the
config file, with some rare exceptions (such as some YARN code that does
expansion of Hadoop configuration).
But there are a few use cases that don't work well in that situation. For
example, consider {{spark.sql.hive.metastore.jars}}. It references a list of
paths containing the classpath for accessing Hive's metastore. If you're
launching an application in cluster mode, it means that whatever is in the
configuration of the edge node needs to match the configuration of the random
node in the cluster where the driver will actually run.
This would be easily solved if there was a way to reference system properties
or env variables; for example, when YARN launches a container, a bunch of env
variables are set, which could be used to modify that path to match the correct
location on the node.
So I'm proposing a change where config properties can opt-in to use this
variable expansion feature; it's opt-in to avoid breaking existing code (who
knows) and to avoid the extra cost of doing the variable expansion of every
config read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
