[
https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239069#comment-17239069
]
Prashant Sharma commented on SPARK-32223:
-----------------------------------------
One problem, I can think of with driver template is, even though it is possible
to mount a config map, it is not as straight forward to mount it as
SPARK_CONF_DIR. One may have to copy the spark.properties during container
init.
> Support adding a user provided config map.
> ------------------------------------------
>
> Key: SPARK-32223
> URL: https://issues.apache.org/jira/browse/SPARK-32223
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: 3.1.0
> Reporter: Prashant Sharma
> Priority: Major
>
> One of the challenge with this is, spark.properties is not user provided and
> is calculated based on certain factors. So a user provided config map, cannot
> be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow
> augmented with the correct spark.properties.
> Q, Do we support update to config map properties for an already running job?
> Ans: No, since the spark.properties is calculated at the time of job
> submission, it cannot be updated on the fly and it is not supported by Spark
> at the moment for all the configuration values.
> Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map
> helps?
> One of the use case, I can think of is programmatically submitting a `spark
> on k8s` job - e.g. spark as a service on a cloud deployment may find this
> feature useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
