Re: [PR] [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories [spark]

2023-12-11 Thread via GitHub
vicennial commented on code in PR #44109: URL: https://github.com/apache/spark/pull/44109#discussion_r1423128412 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -590,6 +590,48 @@ class SparkSession private[sql] ( @Experimental

Re: [PR] [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories [spark]

2023-12-11 Thread via GitHub
hvanhovell commented on code in PR #44109: URL: https://github.com/apache/spark/pull/44109#discussion_r1423125577 ## connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala: ## @@ -108,6 +104,56 @@ class ArtifactManager( */ def

Re: [PR] [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories [spark]

2023-12-11 Thread via GitHub
hvanhovell commented on code in PR #44109: URL: https://github.com/apache/spark/pull/44109#discussion_r1423124563 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -590,6 +590,48 @@ class SparkSession private[sql] ( @Experimental

Re: [PR] [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories [spark]

2023-12-08 Thread via GitHub
vicennial commented on PR #44109: URL: https://github.com/apache/spark/pull/44109#issuecomment-1847023164 cc @hvanhovell @nija-at -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories [spark]

2023-12-08 Thread via GitHub
vicennial commented on PR #44109: URL: https://github.com/apache/spark/pull/44109#issuecomment-1847013631 I've updated the APIs to be simpler to make them simpler to use (String-based source/target). It is now easier to reason about what the directory-strcuture would look like. -- This