Jackey Lee created SPARK-31142:
----------------------------------
Summary: Remove useless conf set in pyspark context
Key: SPARK-31142
URL: https://issues.apache.org/jira/browse/SPARK-31142
Project: Spark
Issue Type: Improvement
Components: PySpark, Spark Core
Affects Versions: 3.0.0, 3.1.0
Reporter: Jackey Lee
In Pyspark/context.py, we extract the configuration of the "spark.executorEnv"
prefix from conf and put it in env. But in PythonWorkerFactory, we have already
obtained these environment variables from ProgressBuilder, thus we do not need
the extra copy.
In addition, in some special environment deployments, user-configured
environment variables such as "spark.executorEnv.Path=/usr/lib:$PATH" will be
automatically parsed as actual machine confs. If we still directly copy and
overwrite it, the configuration will be a wrong config.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
