Lisa Cao created SPARK-60365:
--------------------------------
Summary: spark-submit --remote does not pass the connection string
to Python applications unless spark.api.mode=connect
Key: SPARK-60365
URL: https://issues.apache.org/jira/browse/SPARK-60365
Project: Spark
Issue Type: Bug
Components: PySpark, Connect
Affects Versions: 4.2.0, 4.1.3, 4.3.0
Reporter: Lisa Cao
Running a Python application through {{spark-submit --remote sc://host:port
app.py}} fails on a stock distribution:
{code}
pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_NOT_SET] A
master URL must be set in your configuration.
{code}
The application contains only {{SparkSession.builder.getOrCreate()}}.
{{--remote}} is accepted by {{SparkSubmitArguments}} and lands in
{{spark.remote}}, but {{PythonRunner}} decides how to launch the Python process
from {{spark.api.mode}}, whose default is {{classic}}:
{code:scala}
val apiMode = sparkConf.get(SPARK_API_MODE).toLowerCase(Locale.ROOT)
val isAPIModeClassic = apiMode == "classic"
...
if (sparkConf.getOption("spark.remote").isEmpty || isAPIModeClassic) {
gatewayServer = Some(new Py4JServer(sparkConf)) // classic launch
...
}
...
if (isAPIModeClassic) {
sparkConf.getOption("spark.master").foreach(url => env.put("MASTER", url))
} else {
sparkConf.getOption("spark.remote").foreach(url => env.put("SPARK_REMOTE",
url))
}
{code}
With the default API mode the runner starts a Py4J gateway and exports
{{MASTER}} (unset, because {{--remote}} and {{--master}} are mutually
exclusive) instead of {{SPARK_REMOTE}}. The Python side then builds a classic
session and fails.
Reproduced with the 4.3.0 binary distribution against a Spark Connect server on
a different port, and inside the {{apache/spark:4.2.0}} and
{{apache/spark:4.1.3}} images against their own server.
Forms that work:
* {{spark-submit --remote sc://host:port --conf spark.api.mode=connect app.py}}
* {{SPARK_CONNECT_MODE=1 spark-submit --remote sc://host:port app.py}}
* {{SPARK_REMOTE=sc://host:port spark-submit app.py}} (the Python process
inherits the variable)
* {{spark-submit --remote sc://host:port --class Main app.jar}} (JVM
applications are unaffected)
Expected: {{--remote}} alone should launch the Python application as a Spark
Connect client, as the {{--remote}} help text ({{URL to connect to the server
for Spark Connect}}) describes. Proposed fix: in {{PythonRunner}}, treat a set
{{spark.remote}} as Connect mode regardless of {{spark.api.mode}} (skip the
gateway, export {{SPARK_REMOTE}}), or reject the combination with a clear error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
