HyukjinKwon commented on code in PR #58696:
URL: https://github.com/apache/spark/pull/58696#discussion_r3985160589
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -2787,6 +2787,16 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val STANDALONE_SUBMIT_FILTER_ENVIRONMENT =
+ ConfigBuilder("spark.standalone.submit.filterEnvironment")
Review Comment:
Since this flips a default in a minor release (standalone cluster mode now
forwards only `SPARK_*` vars to the driver, previously the submitter's full
env), it's worth a `docs/core-migration-guide.md` entry so upgrading users
discover the behavior change and the opt-out. The existing 4.3 entries are the
pattern -- add a `## Upgrading from Core 4.3 to 4.4` section with something
like: "Since Spark 4.4.0, in standalone cluster mode the client forwards only
Spark-related environment variables to the driver. To restore the legacy
behavior, set `spark.standalone.submit.filterEnvironment` to `false`." The
config doc here and in `spark-standalone.md` cover the *what*; the migration
guide is where the *change* on upgrade is flagged.
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -2787,6 +2787,16 @@ package object config {
.booleanConf
.createWithDefault(false)
+ private[spark] val STANDALONE_SUBMIT_FILTER_ENVIRONMENT =
+ ConfigBuilder("spark.standalone.submit.filterEnvironment")
+ .doc("In standalone cluster mode, controls whether the client forwards
only " +
+ "Spark-related environment variables to the driver, matching the REST
submission " +
+ "client. If set to false, the full environment of the submitting
process is " +
+ "forwarded to the driver.")
+ .version("4.4.0")
Review Comment:
Should it be 4.3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
