[
https://issues.apache.org/jira/browse/SPARK-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043124#comment-16043124
]
Lantao Jin commented on SPARK-21023:
------------------------------------
[~vanzin] I suggest to change the current behavior and offer a document to
illustrate this. --properties-file will overwrite the args which are set in
spark-defaults.conf first. It's equivalent to set dozens of {{--conf k=v}} in
command line. Please review and open for any ideas.
> Ignore to load default properties file is not a good choice from the
> perspective of system
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-21023
> URL: https://issues.apache.org/jira/browse/SPARK-21023
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 2.1.1
> Reporter: Lantao Jin
> Priority: Minor
>
> The default properties file {{spark-defaults.conf}} shouldn't be ignore to
> load even though the submit arg {{--properties-file}} is set. The reasons are
> very easy to see:
> * Infrastructure team need continually update the {{spark-defaults.conf}}
> when they want set something as default for entire cluster as a tuning
> purpose.
> * Application developer only want to override the parameters they really want
> rather than others they even doesn't know (Set by infrastructure team).
> * The purpose of using {{\-\-properties-file}} from most of application
> developers is to avoid setting dozens of {{--conf k=v}}. But if
> {{spark-defaults.conf}} is ignored, the behaviour becomes unexpected finally.
> All this caused by below codes:
> {code}
> private Properties loadPropertiesFile() throws IOException {
> Properties props = new Properties();
> File propsFile;
> if (propertiesFile != null) {
> // default conf property file will not be loaded when app developer use
> --properties-file as a submit args
> propsFile = new File(propertiesFile);
> checkArgument(propsFile.isFile(), "Invalid properties file '%s'.",
> propertiesFile);
> } else {
> propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE);
> }
> //...
> return props;
> }
> {code}
> I can offer a patch to fix it if you think it make sense.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
