Nick Herzig created SPARK-50126:
-----------------------------------
Summary: PySpark expr() (expression) SQL Function returns None in
Spark Connect
Key: SPARK-50126
URL: https://issues.apache.org/jira/browse/SPARK-50126
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 3.5.3, 3.5.2, 3.5.1, 3.5.0
Environment: Spark 3.5.3
Python 3.9.20
Reporter: Nick Herzig
Fix For: 3.5.4
When PySpark is launched with remote Spark Connect configuration, calls to
pyspark.sql.functions.expr always return `Column<None>`.
PySpark without Spark Connect
{code:java}
$ pyspark
...
>>> from pyspark.sql.functions import expr
>>> e = expr("*")
>>> print(e)
Column<'unresolvedstar()'>
{code}
PySpark with Spark Connect
{code:java}
$ pyspark --remote "sc://localhost:15002"
...
Client connected to the Spark Connect server at localhost:15002
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import expr
>>> e = expr("*")
>>> print(e)
Column<'None'>
{code}
Additional example
{code:java}
$ pyspark
...
Spark context available as 'sc' (master = local[*], app id =
local-1729876117745).
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import expr
>>> expr("*")
Column<'unresolvedstar()'>
>>> SparkSession.builder.master("local[*]").getOrCreate().stop()
>>> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
>>> expr("*")
Column<'None'> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
