[
https://issues.apache.org/jira/browse/SPARK-60365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128918#comment-18128918
]
Lisa Cao commented on SPARK-60365:
----------------------------------
The forms that work are documented in the Spark Connect programming guide
proposed in https://github.com/apache/spark/pull/59613 (the "Where a scheduler
expects spark-submit" section). A fix in PythonRunner can follow separately.
> spark-submit --remote does not pass the connection string to Python
> applications unless spark.api.mode=connect
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60365
> URL: https://issues.apache.org/jira/browse/SPARK-60365
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 4.1.3
> Reporter: Lisa Cao
> Priority: Major
>
> Running a Python application through {{spark-submit --remote sc://host:port
> app.py}} fails on a stock distribution:
> {code}
> pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_NOT_SET] A
> master URL must be set in your configuration.
> {code}
> The application contains only {{SparkSession.builder.getOrCreate()}}.
> {{--remote}} is accepted by {{SparkSubmitArguments}} and lands in
> {{spark.remote}}, but {{PythonRunner}} decides how to launch the Python
> process from {{spark.api.mode}}, whose default is {{classic}}:
> {code:scala}
> val apiMode = sparkConf.get(SPARK_API_MODE).toLowerCase(Locale.ROOT)
> val isAPIModeClassic = apiMode == "classic"
> ...
> if (sparkConf.getOption("spark.remote").isEmpty || isAPIModeClassic) {
> gatewayServer = Some(new Py4JServer(sparkConf)) // classic launch
> ...
> }
> ...
> if (isAPIModeClassic) {
> sparkConf.getOption("spark.master").foreach(url => env.put("MASTER", url))
> } else {
> sparkConf.getOption("spark.remote").foreach(url => env.put("SPARK_REMOTE",
> url))
> }
> {code}
> With the default API mode the runner starts a Py4J gateway and exports
> {{MASTER}} (unset, because {{--remote}} and {{--master}} are mutually
> exclusive) instead of {{SPARK_REMOTE}}. The Python side then builds a classic
> session and fails.
> Reproduced with the 4.3.0 binary distribution against a Spark Connect server
> on a different port, and inside the {{apache/spark:4.2.0}} and
> {{apache/spark:4.1.3}} images against their own server.
> Forms that work:
> * {{spark-submit --remote sc://host:port --conf spark.api.mode=connect
> app.py}}
> * {{SPARK_CONNECT_MODE=1 spark-submit --remote sc://host:port app.py}}
> * {{SPARK_REMOTE=sc://host:port spark-submit app.py}} (the Python process
> inherits the variable)
> * {{spark-submit --remote sc://host:port --class Main app.jar}} (JVM
> applications are unaffected)
> Expected: {{--remote}} alone should launch the Python application as a Spark
> Connect client, as the {{--remote}} help text ({{URL to connect to the server
> for Spark Connect}}) describes. Proposed fix: in {{PythonRunner}}, treat a
> set {{spark.remote}} as Connect mode regardless of {{spark.api.mode}} (skip
> the gateway, export {{SPARK_REMOTE}}), or reject the combination with a clear
> error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
