[
https://issues.apache.org/jira/browse/SPARK-60855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao reassigned SPARK-60855:
--------------------------------
Assignee: Tian Gao
> Remove the Spark JAR from the Python worker PYTHONPATH
> ------------------------------------------------------
>
> Key: SPARK-60855
> URL: https://issues.apache.org/jira/browse/SPARK-60855
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Tian Gao
> Assignee: Tian Gao
> Priority: Major
> Labels: pull-request-available
>
> PythonUtils.sparkPythonPath adds SparkContext.jarOfObject(this), i.e. the
> spark-core JAR, to the PYTHONPATH of every Python process Spark launches
> (Python workers, PythonRunner, Python UDF workers).
> That entry dates from SPARK-1549 (Spark 1.0), when SPARK-1004 packaged
> pyspark/*.py and py4j into the core/assembly JAR so PySpark on YARN could
> import them from the JAR. SPARK-6869 started shipping pyspark.zip and the
> py4j zip to YARN separately, and SPARK-7485 (Spark 1.4) removed the Python
> files from the JAR, but the PYTHONPATH entry was never removed. No Spark
> build has put Python sources into a JAR since 1.4 (the spark-core 4.0.0 JAR
> has 5,261 entries and no .py/.pyc files).
> The entry is not free. Python creates a zipimporter for it and reads the
> JAR's central directory, and every top-level import that misses pyspark.zip
> and py4j probes it. Python workers also call importlib.invalidate_caches()
> when they add Spark files to sys.path; that re-reads the JAR directory once
> per new worker, on Python 3.11-3.12 directly and on 3.13+ lazily through the
> importlib.metadata import done by invalidate_caches(). In setup_spark_files
> this costs about 7-15 ms per new worker (e.g. 13.5 ms -> 4.9 ms on Python
> 3.13).
> Remove the entry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
