[ 
https://issues.apache.org/jira/browse/SPARK-60893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Gao updated SPARK-60893:
-----------------------------
    Affects Version/s: 4.4.0
                           (was: 5.0.0)

> Keep the import caches of Spark's own zip archives when Python workers 
> invalidate import caches
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60893
>                 URL: https://issues.apache.org/jira/browse/SPARK-60893
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.4.0
>            Reporter: Tian Gao
>            Assignee: Tian Gao
>            Priority: Major
>              Labels: pull-request-available
>
> Python workers call importlib.invalidate_caches() in 
> worker_util.setup_spark_files when they add Spark files to sys.path, which 
> happens on the first task of every new worker process (and on every task when 
> spark.python.worker.reuse is disabled).
> importlib.invalidate_caches() calls invalidate_caches() on every finder in 
> sys.path_importer_cache. Each package imported from a zip archive has its own 
> zipimporter there (pyspark.zip has about 15 after importing pyspark.worker). 
> Up to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's 
> whole central directory, so pyspark.zip is re-read once per package imported 
> from it. On 3.13+ the listing is dropped and re-read once per archive on the 
> next lookup, which happens right away because invalidate_caches() imports 
> importlib.metadata.
> Spark's own archives, the zips the worker's pyspark and py4j were loaded from 
> ($SPARK_HOME/python/lib/pyspark.zip and the py4j zip, or the copies Spark 
> ships to YARN containers), are not replaced while an application runs, so 
> their listings never need to be refreshed. Keep the zipimporters of those 
> archives when invalidating, and keep invalidating everything else, including 
> user archives on PYTHONPATH (which may be rebuilt while the Python daemon 
> lives) and zips added through python_includes (addPyFile, --py-files, Spark 
> Connect artifacts).
> With the spark-core JAR removed from the worker path (SPARK-60855), 
> setup_spark_files goes from about 40-42 ms to about 5 ms on Python 3.11/3.12, 
> and from about 5 ms to about 2.3 ms on 3.13/3.14.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to