This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new c225d4aefef [SPARK-44829][CONNECT] Expose uploadAllArtifactClasses in
ArtifactManager to `sql` package
c225d4aefef is described below
commit c225d4aefef87fd029b0bcbaa23385ce370f0324
Author: Vsevolod Stepanov <vsevolod.stepa...@databricks.com>
AuthorDate: Wed Aug 16 21:05:41 2023 +0200
[SPARK-44829][CONNECT] Expose uploadAllArtifactClasses in ArtifactManager
to `sql` package
### What changes were proposed in this pull request?
This PR exposes `SparkConnectClient`'s
`artifactManager.uploadAllArtifactClasses` to `sql` package.
### Why are the changes needed?
Currently, UDF classfiles are uploaded in all analyze/execute operations.
However, if codepath used to call Scala UDF bypasses we'll need to manually
call this method to make sure that all classfiles are uploaded
### How was this patch tested?
No tests needed - this PR only exposes a method to sql package
Closes #42516 from vsevolodstep-db/SPARK-44829/uploadAllArtifactClasses.
Authored-by: Vsevolod Stepanov <vsevolod.stepa...@databricks.com>
Signed-off-by: Herman van Hovell <her...@databricks.com>
(cherry picked from commit 6ae4f4f751fe74944f3acd29c836f0100812117b)
Signed-off-by: Herman van Hovell <her...@databricks.com>
---
.../org/apache/spark/sql/connect/client/SparkConnectClient.scala | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
index 637499f090c..c41f6dfaae1 100644
---
a/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
+++
b/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala
@@ -62,6 +62,12 @@ private[sql] class SparkConnectClient(
new ArtifactManager(configuration, sessionId, bstub, stub)
}
+ /**
+ * Manually triggers upload of all classfile artifacts to the Spark Connect
Server
+ */
+ private[sql] def uploadAllClassFileArtifacts(): Unit =
+ artifactManager.uploadAllClassFileArtifacts()
+
/**
* Dispatch the [[proto.AnalyzePlanRequest]] to the Spark Connect server.
* @return
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
