[
https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489186#comment-16489186
]
Trevor McKay commented on SPARK-24091:
--------------------------------------
I had a similar situation in a project.
One way to handle this is allow a user-specified ConfigMap to be mounted at
another location, then in the image startup script (entrypoint.sh) check for a
non-empty directory. If there are files there, copy them one by one to
$SPARK_HOME/conf. This gives you an override semantic for just those files
included. The ConfigMap volume can be marked as optional so that if it's not
created the container will still start.
Another way to do this of course would be to somehow allow the user to modify
the contents of the original ConfigMap mentioned above, but this might be tough
depending on how it's named, when it's generated, etc etc.
> Internally used ConfigMap prevents use of user-specified ConfigMaps carrying
> Spark configs files
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-24091
> URL: https://issues.apache.org/jira/browse/SPARK-24091
> Project: Spark
> Issue Type: Brainstorming
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Yinan Li
> Priority: Major
>
> The recent PR [https://github.com/apache/spark/pull/20669] for removing the
> init-container introduced a internally used ConfigMap carrying Spark
> configuration properties in a file for the driver. This ConfigMap gets
> mounted under {{$SPARK_HOME/conf}} and the environment variable
> {{SPARK_CONF_DIR}} is set to point to the mount path. This pretty much
> prevents users from mounting their own ConfigMaps that carry custom Spark
> configuration files, e.g., {{log4j.properties}} and {{spark-env.sh}} and
> leaves users with only the option of building custom images. IMO, it is very
> useful to support mounting user-specified ConfigMaps for custom Spark
> configuration files. This worths further discussions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
