[
https://issues.apache.org/jira/browse/SPARK-60012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qianlan Chen updated SPARK-60012:
---------------------------------
Description:
SPARK-59553 added per-session Maven resolution to ArtifactManager. The resolver
stores its Ivy cache at {{{}artifactPath/.ivy-cache{}}}.
{{ArtifactManager.clone()}} recursively copies the entire parent
{{{}artifactPath{}}}. After Maven resolution, this copies both:
* The disposable Ivy cache, including downloaded and retrieved jars.
* The installed session artifacts under {{{}jars/{}}}, {{{}classes/{}}}, and
other artifact directories.
For a large transitive dependency tree, every initialized cloned session can
therefore incur unnecessary disk usage and clone latency.
h2. Proposed change
Update {{ArtifactManager.clone()}} to exclude the top-level {{.ivy-cache}}
directory while continuing to copy installed session artifacts.
The cloned session should lazily create a fresh {{.ivy-cache}} if it later
performs Maven resolution.
h2. Acceptance criteria
* Installed jars, classes, and Python artifacts remain available in the
cloned session.
* The parent's {{.ivy-cache}} is not copied.
* Maven resolution in the cloned session still succeeds and creates a new
cache when needed.
* Add a regression test covering these behaviors.
was:
SPARK-59553 added per-session Maven resolution to ArtifactManager. The resolver
stores its Ivy cache at \{{artifactPath/.ivy-cache}}.
\{{ArtifactManager.clone()}} recursively copies the entire parent
\{{artifactPath}}. After Maven resolution, this copies both:
* The disposable Ivy cache, including downloaded and retrieved jars.
* The installed session artifacts under \{{jars/}}, \{{classes/}}, and other
artifact directories.
For a large transitive dependency tree, every initialized cloned session can
therefore incur unnecessary disk usage and clone latency.
h2. Proposed change
Update \{{ArtifactManager.clone()}} to exclude the top-level \{{.ivy-cache}}
directory while continuing to copy installed session artifacts.
The cloned session should lazily create a fresh \{{.ivy-cache}} if it later
performs Maven resolution.
h2. Acceptance criteria
* Installed jars, classes, and Python artifacts remain available in the
cloned session.
* The parent's \{{.ivy-cache}} is not copied.
* Maven resolution in the cloned session still succeeds and creates a new
cache when needed.
* Add a regression test covering these behaviors.
h2. Related work
* SPARK-59553
> Avoid copying Ivy caches when cloning ArtifactManager
> -----------------------------------------------------
>
> Key: SPARK-60012
> URL: https://issues.apache.org/jira/browse/SPARK-60012
> Project: Spark
> Issue Type: Bug
> Components: Connect, SQL
> Affects Versions: 4.4.0
> Reporter: Qianlan Chen
> Priority: Major
>
> SPARK-59553 added per-session Maven resolution to ArtifactManager. The
> resolver stores its Ivy cache at {{{}artifactPath/.ivy-cache{}}}.
> {{ArtifactManager.clone()}} recursively copies the entire parent
> {{{}artifactPath{}}}. After Maven resolution, this copies both:
> * The disposable Ivy cache, including downloaded and retrieved jars.
> * The installed session artifacts under {{{}jars/{}}}, {{{}classes/{}}},
> and other artifact directories.
> For a large transitive dependency tree, every initialized cloned session can
> therefore incur unnecessary disk usage and clone latency.
> h2. Proposed change
> Update {{ArtifactManager.clone()}} to exclude the top-level {{.ivy-cache}}
> directory while continuing to copy installed session artifacts.
> The cloned session should lazily create a fresh {{.ivy-cache}} if it later
> performs Maven resolution.
> h2. Acceptance criteria
> * Installed jars, classes, and Python artifacts remain available in the
> cloned session.
> * The parent's {{.ivy-cache}} is not copied.
> * Maven resolution in the cloned session still succeeds and creates a new
> cache when needed.
> * Add a regression test covering these behaviors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
