dongjoon-hyun commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579602687
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##########
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
this
}
+ // These submit configuration only effect when config before submit app.
+ private val SUBMIT_LAUNCHER_CONFIG =
+ Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+ DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS,
DRIVER_EXTRA_LIBRARY_PATH,
+ PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL,
CHILD_PROCESS_LOGGER_NAME,
+ CHILD_CONNECTION_TIMEOUT)
+
+ // These configuration can effect when SparkContext is not started.
+ private val EXECUTOR_LAUNCHER_CONFIG =
+ Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH,
EXECUTOR_DEFAULT_JAVA_OPTIONS,
+ EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH,
EXECUTOR_CORES)
+
+ def checkAndSetConfig(key: String, value: String): Unit = {
+ if (SparkContext.getActive.isEmpty) {
+ if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+ logWarning(s"Since spark has been started, configuration ${key}
won't work" +
+ s" when set it here")
Review comment:
Can we have a directional guide instead of `xxx won't work when set it
here`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
