[
https://issues.apache.org/jira/browse/SPARK-54313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-54313.
------------------------------------
Fix Version/s: 4.2.0
Target Version/s: (was: 4.2.0)
Assignee: John Zhuge
Resolution: Fixed
Issue resolved by https://github.com/apache/spark/pull/53012
> Add --extra-properties-file option for configuration layering in spark-submit
> -----------------------------------------------------------------------------
>
> Key: SPARK-54313
> URL: https://issues.apache.org/jira/browse/SPARK-54313
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.0.1
> Reporter: John Zhuge
> Assignee: John Zhuge
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> This proposal adds support for specifying multiple properties files in
> spark-submit through a repeatable {{--extra-properties-file}} option. This
> enables configuration layering and composition, similar to SPARK-34345 but a
> bit more cleaner.
> Large-scale Spark deployments often need to compose configurations from
> multiple sources, for example:
> * Base/shared defaults - Common settings across all applications
> * Environment-specific configs - Dev/staging/prod differences
> * Cluster-specific settings - Different clusters, regions, or Kubernetes
> namespaces
> * Application-specific overrides - Per-job tuning
> Currently, we must either merge properties files before submission or rely
> solely on {{--conf}} flags (command line can get very long).
> Proposed command-line syntax:
> {code:bash}
> spark-submit \
> --properties-file /path/to/base.conf \
> --load-spark-defaults \
> --extra-properties-file /path/to/override1.conf \
> --extra-properties-file /path/to/override2.conf \
> --conf spark.executor.memory=4g \
> --class com.example.MyApp \
> myapp.jar
> {code}
> Property precedence (highest to lowest):
> # {{--conf}} flags (command-line overrides)
> # {{--extra-properties-file}} files (in order specified, later overrides
> earlier)
> # {{--properties-file}} (base file)
> # {{conf/spark-defaults.conf}} (when {{\-\-properties-file}} is not
> specified or if {{--load-spark-defaults}} is used)
> Key behaviors:
> - {{--extra-properties-file}} can be specified multiple times
> - Files are loaded left-to-right; later values override earlier ones
> - Missing files cause validation errors
> - Works with all deployment modes (client, cluster, local)
> - Compatible with {{--load-spark-defaults}} flag
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
