[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232631#comment-17232631 ] Prashant Sharma edited comment on SPARK-30985 at 11/16/20, 9:21 AM: Reopening this JIRA as this was closed because, I created a PR incorrectly targeting the umbrella JIRA instead of the subtask : was (Author: prashant_): Reopening this JIRA as, this is Umbrella jira and I created a PR incorrectly targeting the umbrella JIRA instead of the subtask : > Propagate SPARK_CONF_DIR files to driver and exec pods. > --- > > Key: SPARK-30985 > URL: https://issues.apache.org/jira/browse/SPARK-30985 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Priority: Major > Fix For: 3.1.0 > > > SPARK_CONF_DIR hosts configuration files like, > 1) spark-defaults.conf - containing all the spark properties. > 2) log4j.properties - Logger configuration. > 3) spark-env.sh - Environment variables to be setup at driver and executor. > 4) core-site.xml - Hadoop related configuration. > 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. > 6) metrics.properties - Spark metrics. > 7) Any user specific - library or framework specific configuration file. > Traditionally, SPARK_CONF_DIR has been the home to all user specific > configuration files. > So this feature, will let the user specific configuration files be mounted on > the driver and executor pods' SPARK_CONF_DIR. > Please review the attached design doc, for more details. > > [Google docs link|https://bit.ly/spark-30985] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232601#comment-17232601 ] Dongjoon Hyun edited comment on SPARK-30985 at 11/16/20, 8:14 AM: -- [~prashant]. You should not make a PR with the umbrella JIRA. The umbrella Jira should be resolved when the all subtask JIRAs are resolved. However, if you make a PR with the umbrella Jira ID, Spark merge script resolves it. Since this seems to be created long time ago, the Jira management is up to you. The above comment is just my suggestion. was (Author: dongjoon): [~prashant]. You should not make a PR with the umbrella JIRA. The umbrella Jira is resolved when the all subtask JIRAs are resolved. If you make a PR with the umbrella Jira ID, Spark merge script resolves it. Since this seems to be created long time ago, the Jira management is up to you. The above comment is just my suggestion. > Propagate SPARK_CONF_DIR files to driver and exec pods. > --- > > Key: SPARK-30985 > URL: https://issues.apache.org/jira/browse/SPARK-30985 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Assignee: Prashant Sharma >Priority: Major > Fix For: 3.1.0 > > > SPARK_CONF_DIR hosts configuration files like, > 1) spark-defaults.conf - containing all the spark properties. > 2) log4j.properties - Logger configuration. > 3) spark-env.sh - Environment variables to be setup at driver and executor. > 4) core-site.xml - Hadoop related configuration. > 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. > 6) metrics.properties - Spark metrics. > 7) Any user specific - library or framework specific configuration file. > Traditionally, SPARK_CONF_DIR has been the home to all user specific > configuration files. > So this feature, will let the user specific configuration files be mounted on > the driver and executor pods' SPARK_CONF_DIR. > Please review the attached design doc, for more details. > > [Google docs link|https://bit.ly/spark-30985] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232600#comment-17232600 ] Dongjoon Hyun edited comment on SPARK-30985 at 11/16/20, 8:08 AM: -- ~It's a little weird to convert a merged JIRA into an umbrella.~ Hmm. It seems that I was confused the history. was (Author: dongjoon): It's a little weird to convert a merged JIRA into an umbrella. > Propagate SPARK_CONF_DIR files to driver and exec pods. > --- > > Key: SPARK-30985 > URL: https://issues.apache.org/jira/browse/SPARK-30985 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Assignee: Prashant Sharma >Priority: Major > Fix For: 3.1.0 > > > SPARK_CONF_DIR hosts configuration files like, > 1) spark-defaults.conf - containing all the spark properties. > 2) log4j.properties - Logger configuration. > 3) spark-env.sh - Environment variables to be setup at driver and executor. > 4) core-site.xml - Hadoop related configuration. > 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. > 6) metrics.properties - Spark metrics. > 7) Any user specific - library or framework specific configuration file. > Traditionally, SPARK_CONF_DIR has been the home to all user specific > configuration files. > So this feature, will let the user specific configuration files be mounted on > the driver and executor pods' SPARK_CONF_DIR. > Please review the attached design doc, for more details. > > [Google docs link|https://bit.ly/spark-30985] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232600#comment-17232600 ] Dongjoon Hyun edited comment on SPARK-30985 at 11/16/20, 8:08 AM: -- -It's a little weird to convert a merged JIRA into an umbrella.- Hmm. It seems that I was confused the history. was (Author: dongjoon): ~It's a little weird to convert a merged JIRA into an umbrella.~ Hmm. It seems that I was confused the history. > Propagate SPARK_CONF_DIR files to driver and exec pods. > --- > > Key: SPARK-30985 > URL: https://issues.apache.org/jira/browse/SPARK-30985 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.1.0 >Reporter: Prashant Sharma >Assignee: Prashant Sharma >Priority: Major > Fix For: 3.1.0 > > > SPARK_CONF_DIR hosts configuration files like, > 1) spark-defaults.conf - containing all the spark properties. > 2) log4j.properties - Logger configuration. > 3) spark-env.sh - Environment variables to be setup at driver and executor. > 4) core-site.xml - Hadoop related configuration. > 5) fairscheduler.xml - Spark's fair scheduling policy at the job level. > 6) metrics.properties - Spark metrics. > 7) Any user specific - library or framework specific configuration file. > Traditionally, SPARK_CONF_DIR has been the home to all user specific > configuration files. > So this feature, will let the user specific configuration files be mounted on > the driver and executor pods' SPARK_CONF_DIR. > Please review the attached design doc, for more details. > > [Google docs link|https://bit.ly/spark-30985] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
