Xi Lyu created SPARK-52104:
------------------------------
Summary: Validate column name eagerly in Spark Connect Scala Client
Key: SPARK-52104
URL: https://issues.apache.org/jira/browse/SPARK-52104
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.0.0, 4.1.0
Reporter: Xi Lyu
Currently, calling DataFrame.col(colName) with a non-existent column in the
Scala Spark Connect client does not raise an error. In contrast, both PySpark
(in Spark Connect and Spark Classic) and Scala in Spark Classic do raise an
exception in such cases. This leads to inconsistent behavior between Spark
Connect and Spark Classic in Scala.
In this task, eager validation of column names should be implemented in the
DataFrame.col(colName) method of the Scala client to ensure consistent behavior
with both Spark Classic and PySpark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
