[jira] [Created] (SPARK-51789) Respect spark.api.mode and spark.remote properly when parsing arguments in Spark Submission

[Hyukjin Kwon (Jira)]

Hyukjin Kwon created SPARK-51789:
------------------------------------

             Summary: Respect spark.api.mode and spark.remote properly when 
parsing arguments in Spark Submission
                 Key: SPARK-51789
                 URL: https://issues.apache.org/jira/browse/SPARK-51789
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.0.0
            Reporter: Hyukjin Kwon


{code}
SPARK_CONNECT_MODE=connect ./bin/pyspark --conf spark.api.mode=classic --master 
local --verbose
{code}

Fails as below:

{code}
/.../spark/python/pyspark/shell.py:94: UserWarning: Failed to initialize Spark 
session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../spark/python/pyspark/shell.py", line 89, in <module>
    spark = SparkSession._create_shell_session()
  File "/.../spark/python/pyspark/sql/session.py", line 1249, in 
_create_shell_session
    return SparkSession._getActiveSessionOrCreate()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/.../spark/python/pyspark/sql/session.py", line 1260, in 
_getActiveSessionOrCreate
    spark = SparkSession.getActiveSession()
  File "/.../spark/python/pyspark/sql/utils.py", line 357, in wrapped
    return f(*args, **kwargs)
  File "/.../spark/python/pyspark/sql/session.py", line 747, in getActiveSession
    if jSparkSessionClass.getActiveSession().isDefined():
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
TypeError: 'JavaPackage' object is not callable
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org