qianlan717 opened a new pull request, #59196: URL: https://github.com/apache/spark/pull/59196
### What changes were proposed in this pull request? This PR makes runtime Maven artifact resolution actively cancellable after execution has entered `Ivy.resolve`. The resolution runs on a dedicated worker thread while the caller polls its cancellation signal. On cancellation, Spark closes the per-resolution Ivy HTTP handler, interrupts the active Ivy instance, waits for the worker to finish, restores the previous URL handler, and releases the global Ivy lock. It also adds coverage for cancellation during a blocked network read, resolver timeouts with custom Ivy settings, server-side resolution using driver Ivy settings, and atomic artifact registration after resolution failures. ### Why are the changes needed? Spark Connect already propagated the AddArtifacts gRPC cancellation state into Maven resolution. However, after Ivy entered a blocking network read inside `Ivy.resolve`, that state was no longer checked. A cancelled request could therefore retain a server thread, the global Ivy lock, and its repository connection until the configured read timeout expired. ### Does this PR introduce _any_ user-facing change? Yes. Cancelling a Spark Connect AddArtifacts request during server-side Maven resolution now stops the active Ivy resolution promptly instead of waiting for the repository read timeout. There are no API changes. ### How was this patch tested? Added a `MavenUtilsSuite` test backed by a server that accepts a connection but never responds. The test configures a 30-second network timeout, requests cancellation, verifies a `CancellationException` within 5 seconds, and performs a second resolution to confirm that the Ivy worker exited and released the global lock. Added Spark Connect handler tests for driver Ivy settings and failure atomicity. Built the Spark distribution with Java 17 using: ``` MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com \ sbt_jar=/usr/share/sbt/bin/sbt-launch.jar \ ./build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Phive \ 'set ThisBuild / dependencyOverrides += "at.yawk.lz4" % "lz4-java" % "1.11.2"' \ package ``` Also tested manually with a Spark Connect server and a local repository endpoint that accepts connections without responding. Closing the client produced an immediate server-side `CancellationException`, closed the repository socket, and removed the blocked Ivy resolution thread. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
