Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-23 Thread via GitHub
HyukjinKwon commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1689051900 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -890,7 +891,49 @@ object SparkSession extends Logging { // the re

Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-23 Thread via GitHub
HyukjinKwon commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1689052244 ## core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala: ## @@ -74,6 +77,16 @@ object PythonRunner { // Launch Python process val builder = new

Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-23 Thread via GitHub
hvanhovell commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1689030629 ## core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala: ## @@ -74,6 +77,16 @@ object PythonRunner { // Launch Python process val builder = new P

Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-23 Thread via GitHub
hvanhovell commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1689027244 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -890,7 +891,49 @@ object SparkSession extends Logging { // the rem

Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-23 Thread via GitHub
hvanhovell commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1689027244 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -890,7 +891,49 @@ object SparkSession extends Logging { // the rem

Re: [PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-21 Thread via GitHub
HyukjinKwon commented on code in PR #47434: URL: https://github.com/apache/spark/pull/47434#discussion_r1685869245 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala: ## @@ -890,7 +891,57 @@ object SparkSession extends Logging { // the re

[PR] [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect [spark]

2024-07-21 Thread via GitHub
HyukjinKwon opened a new pull request, #47434: URL: https://github.com/apache/spark/pull/47434 ### What changes were proposed in this pull request? This PR proposes to add the support of `--remote` at `bin/spark-submit` so it can use Spark Connect easily. This PR inclues: - Make `b