Tian Gao created SPARK-54744:
--------------------------------
Summary: Only invalidate import cache when necessary in worker
process
Key: SPARK-54744
URL: https://issues.apache.org/jira/browse/SPARK-54744
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Tian Gao
Each importlib.invalidate_caches call could take tens of ms - which is a pretty
significant number. When we reuse workers, this is unnecessary because we are
not introducing any new paths. We should only do it when necessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
