[
https://issues.apache.org/jira/browse/SPARK-59938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59938.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59196
[https://github.com/apache/spark/pull/59196]
> Improve cancellatio for server-side Maven artifact resolution
> -------------------------------------------------------------
>
> Key: SPARK-59938
> URL: https://issues.apache.org/jira/browse/SPARK-59938
> Project: Spark
> Issue Type: Improvement
> Components: Connect, SQL
> Affects Versions: 4.4.0
> Reporter: Qianlan Chen
> Assignee: Qianlan Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> This is follow-up work from SPARK-59553.
> Current behavior
> Server-side Maven artifact resolution checks whether the gRPC context is
> cancelled or the thread is interrupted:
> * before starting resolution;
> * while polling for the global Ivy lock;
> * immediately after acquiring the lock; and
> * after resolution completes.
> Cancellation is not checked while the synchronous ivy.resolve call is
> running. During active resolution, we currently rely on the network-operation
> timeouts:
> * spark.jars.ivyConnectTimeout: 30 seconds by default
> * spark.jars.ivyReadTimeout: 5 minutes by default
> Both timeouts are capped by the remaining RPC deadline. However, these are
> per-network-operation timeouts rather than a total resolution timeout. A
> cancelled request may therefore retain the global Ivy lock until resolution
> completes or a network operation times out.
> Follow-up work
> 1. Improve cancellation behavior while Ivy resolution is actively running.
> 2. Add coverage for resolution timeouts when custom Ivy settings are used.
> 3. Add an atomicity test where a valid uploaded artifact is followed by a
> failing Maven artifact, verifying that no artifacts are registered.
> 4. Add an end-to-end Spark Connect handler test that exercises Maven
> resolution without using the fake injected artifact.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
