Tian Gao created SPARK-60893:
--------------------------------
Summary: Skip zip archives present at Python worker startup when
invalidating import caches
Key: SPARK-60893
URL: https://issues.apache.org/jira/browse/SPARK-60893
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Tian Gao
Python workers call importlib.invalidate_caches() in
worker_util.setup_spark_files when they add Spark files to sys.path, which
happens on the first task of every new worker process (and on every task when
spark.python.worker.reuse is disabled).
importlib.invalidate_caches() calls invalidate_caches() on every finder in
sys.path_importer_cache. Each package imported from a zip archive has its own
zipimporter there (pyspark.zip has about 15 after importing pyspark.worker). Up
to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's whole
central directory, so pyspark.zip is re-read once per package imported from it.
On 3.13+ the listing is dropped and re-read once per archive on the next
lookup, which happens right away because invalidate_caches() imports
importlib.metadata.
The zip archives that are on sys.path when the worker starts (pyspark.zip, the
py4j zip, and the rest of PYTHONPATH such as YARN-localized --py-files) do not
change while the worker runs, so their listings never need to be refreshed.
Keep the zipimporters of those archives when invalidating, and keep
invalidating everything else, including zips added later through
python_includes (addPyFile, --py-files, Spark Connect artifacts).
setup_spark_files goes from about 40-57 ms to about 5 ms on Python 3.11/3.12,
and from about 13 ms to about 2.3 ms on 3.13/3.14.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
