[jira] [Updated] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-32223: Description: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a `spark on k8s` job - e.g. spark as a service on a cloud deployment may find this feature useful. was: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a spark on k8s job - e.g. spark as a service on cloud deployment may find this feature useful. > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > One of the challenge with this is, spark.properties is not user provided and > is calculated based on certain factors. So a user provided config map, cannot > be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow > augmented with the correct spark.properties. > Q, Do we support update to config map properties for an already running job? > Ans: No, since the spark.properties is calculated at the time of job > submission, it cannot be updated on the fly and it is not supported by Spark > at the moment for all the configuration values. > Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map > helps? > One of the use case, I can think of is programmatically submitting a `spark > on k8s` job - e.g. spark as a service on a cloud deployment may find this > feature useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-32223: Description: One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties. Q, Do we support update to config map properties for an already running job? Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values. Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps? One of the use case, I can think of is programmatically submitting a spark on k8s job - e.g. spark as a service on cloud deployment may find this feature useful. was:The semantics of this will be discussed and added soon. > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > One of the challenge with this is, spark.properties is not user provided and > is calculated based on certain factors. So a user provided config map, cannot > be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow > augmented with the correct spark.properties. > Q, Do we support update to config map properties for an already running job? > Ans: No, since the spark.properties is calculated at the time of job > submission, it cannot be updated on the fly and it is not supported by Spark > at the moment for all the configuration values. > Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map > helps? > One of the use case, I can think of is programmatically submitting a spark on > k8s job - e.g. spark as a service on cloud deployment may find this feature > useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32223) Support adding a user provided config map.
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32223: -- Parent: SPARK-33461 (was: SPARK-30985) > Support adding a user provided config map. > -- > > Key: SPARK-32223 > URL: https://issues.apache.org/jira/browse/SPARK-32223 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > > The semantics of this will be discussed and added soon. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org