[
https://issues.apache.org/jira/browse/SPARK-61771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61771:
-----------------------------------
Labels: pull-request-available (was: )
> Raise the errors of Column.alias when the plan is built, as PySpark's
> ColumnAlias raises them
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-61771
> URL: https://issues.apache.org/jira/browse/SPARK-61771
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Column.alias and Column.name of the drop-in raised every error of their names
> and metadata at the call, where PySpark's ColumnAlias raises them when a
> request builds the plan: a name that protobuf refuses (TypeError, or UTF-8's
> UnicodeEncodeError and UnicodeDecodeError), metadata of one name that
> json.dumps refuses, and metadata of no names (CANNOT_PROVIDE_METADATA). So a
> Column that PySpark builds and never uses raised here, and the Column of such
> an alias did not copy or pickle. Its __repr__ also differed: PySpark's joins
> the names, which raises a TypeError of a name that is not a str (bytes too).
> This raises them as PySpark does: the core expression of such an alias
> carries its errors (ExpressionNode::Unbuildable), which a request raises in
> the order in which PySpark builds the plan, and the __repr__ of the Column
> raises PySpark's TypeError of the join. A pickled or copied Column keeps the
> errors, also on a Python worker of the server.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]