[
https://issues.apache.org/jira/browse/SPARK-60761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60761:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect DataFrame.withColumns({}) fails with an empty AssertionError,
> while classic returns the DataFrame unchanged
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60761
> URL: https://issues.apache.org/jira/browse/SPARK-60761
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 5.0.0
> Reporter: Shuheng Ma
> Priority: Minor
> Labels: pull-request-available
>
> h3. Description
> In Spark Connect, {{DataFrame.withColumns({})}} raises an {{AssertionError}}
> with no message, while classic PySpark returns the DataFrame unchanged.
> An empty dict usually comes from building the column map in code, for example
> by filtering a list of columns where none match. A user on Databricks
> serverless (which uses Spark Connect) reported this:
> https://community.databricks.com/t5/data-engineering/bug-withcolumns-in-pyspark-doesn-t-handle-empty-dictionary/td-p/136875
> h3. Reproducer
> {code:python}
> # spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
> .master("local[2]") for classic
> from pyspark.sql import functions as F
> df = spark.range(3)
> new_cols = {c + "_x": F.col(c) for c in ["missing"] if c in df.columns} # {}
> df.withColumns(new_cols).columns
> # classic: ['id']
> # Connect: AssertionError (no message)
> {code}
> h3. Cause
> {{WithColumns.__init__}} in {{python/pyspark/sql/connect/plan.py}} has
> {{assert len(columnNames) > 0}}:
> https://github.com/apache/spark/blob/1dc0839ca2918ae78eae39635a551648c4e0789f/python/pyspark/sql/connect/plan.py#L688
> The assert was added in SPARK-41936, when {{DataFrame.withMetadata}} (which
> always passes one column) started reusing the {{WithColumns}} plan. The
> server side already supports an empty list:
> {{SparkConnectPlanner.transformWithColumns}} passes it to
> {{Dataset.withColumns}}, which returns the DataFrame unchanged.
> h3. Suggested fix
> Remove that assert, and add the empty dict case to the shared
> {{test_with_columns}} test so both classic and Connect run it.
> Tested on PySpark 4.2.0 and 4.3.0 from PyPI and on master (5.0.0-SNAPSHOT),
> Python 3.12, with a local Spark Connect server (first line of the
> reproducer). I have a fix with tests ready and will open a PR shortly.
> Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
> reproducer and this description.
> Generated-by: Claude Code (Claude Opus 5.5)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
