[
https://issues.apache.org/jira/browse/SPARK-53503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063678#comment-18063678
]
Holden Karau commented on SPARK-53503:
--------------------------------------
I don't think is a great addition, we often use these properties to setup
access to remote filesystems etc so I'm inclined to say no.
> Support remote Properties Files from hdfs in Spark Submit --properties-file
> Option
> ----------------------------------------------------------------------------------
>
> Key: SPARK-53503
> URL: https://issues.apache.org/jira/browse/SPARK-53503
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 4.1.0
> Reporter: Samrat Deb
> Priority: Major
>
> Proposal is to enhance Spark's
> {code:java}
> --properties-file{code}
> option to support remote file systems (HDFS) by automatically downloading
> remote properties files to the local Spark configuration directory before
> loading them.
>
> Currently, Spark's
> {code:java}
> --properties-file{code}
> option only supports local file system paths. When a properties file is
> specified:
> If a user-specified properties file exists locally, it is loaded
> Otherwise, the default
> {code:java}
> spark-defaults.conf{code}
> file from the Spark configuration directory is loaded
> Remote URIs (hdfs:// etc.) cause validation failures in the launcher
> Modify the properties file loading logic to support remote file systems
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
