Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-16 Thread via GitHub
nchammas commented on PR #46364: URL: https://github.com/apache/spark/pull/46364#issuecomment-2116496398 Looks like the docs I am looking for are in #45340. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-12 Thread via GitHub
nchammas commented on PR #46364: URL: https://github.com/apache/spark/pull/46364#issuecomment-2106321926 What's a Spark Connect Plugin? And where can we read some documentation about how to create one? I see there is [this blog post][1] describing how to extend Spark Connect, but I a

Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-07 Thread via GitHub
HyukjinKwon closed pull request #46364: [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect URL: https://github.com/apache/spark/pull/46364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-07 Thread via GitHub
HyukjinKwon commented on PR #46364: URL: https://github.com/apache/spark/pull/46364#issuecomment-2097782732 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-03 Thread via GitHub
dongjoon-hyun commented on PR #46364: URL: https://github.com/apache/spark/pull/46364#issuecomment-2093265227 Could you re-trigger CIs, @tomvanbussel ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-03 Thread via GitHub
HyukjinKwon commented on code in PR #46364: URL: https://github.com/apache/spark/pull/46364#discussion_r1589059935 ## connector/connect/server/src/test/scala/org/apache/spark/sql/connect/plugin/SparkConnectPluginRegistrySuite.scala: ## @@ -20,8 +20,8 @@ package org.apache.spark.

[PR] [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect [spark]

2024-05-03 Thread via GitHub
tomvanbussel opened a new pull request, #46364: URL: https://github.com/apache/spark/pull/46364 ### What changes were proposed in this pull request? This PR enables Spark Connect plugins to process `Relation`s and `Expression`s using `SparkConnectPlanner` on the server side and to cre