Samrat Deb created SPARK-53503:
----------------------------------
Summary: Support remote Properties Files from hdfs in Spark Submit
--properties-file Option
Key: SPARK-53503
URL: https://issues.apache.org/jira/browse/SPARK-53503
Project: Spark
Issue Type: Improvement
Components: Spark Submit
Affects Versions: 4.1.0
Reporter: Samrat Deb
Proposal is to enhance Spark's
{code:java}
--properties-file{code}
option to support remote file systems (HDFS) by automatically downloading
remote properties files to the local Spark configuration directory before
loading them.
Currently, Spark's
{code:java}
--properties-file{code}
option only supports local file system paths. When a properties file is
specified:
If a user-specified properties file exists locally, it is loaded
Otherwise, the default
{code:java}
spark-defaults.conf{code}
file from the Spark configuration directory is loaded
Remote URIs (hdfs:// etc.) cause validation failures in the launcher
Modify the properties file loading logic to support remote file systems
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
