Hyukjin Kwon created SPARK-61771:
------------------------------------
Summary: Raise the errors of Column.alias when the plan is built,
as PySpark's ColumnAlias raises them
Key: SPARK-61771
URL: https://issues.apache.org/jira/browse/SPARK-61771
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Column.alias and Column.name of the drop-in raised every error of their names
and metadata at the call, where PySpark's ColumnAlias raises them when a
request builds the plan: a name that protobuf refuses (TypeError, or UTF-8's
UnicodeEncodeError and UnicodeDecodeError), metadata of one name that
json.dumps refuses, and metadata of no names (CANNOT_PROVIDE_METADATA). So a
Column that PySpark builds and never uses raised here, and the Column of such
an alias did not copy or pickle. Its __repr__ also differed: PySpark's joins
the names, which raises a TypeError of a name that is not a str (bytes too).
This raises them as PySpark does: the core expression of such an alias carries
its errors (ExpressionNode::Unbuildable), which a request raises in the order
in which PySpark builds the plan, and the __repr__ of the Column raises
PySpark's TypeError of the join. A pickled or copied Column keeps the errors,
also on a Python worker of the server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]