[
https://issues.apache.org/jira/browse/SPARK-60948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60948:
-----------------------------------
Labels: pull-request-available (was: )
> Render Column reprs in Spark Connect Rust as PySpark Connect does
> -----------------------------------------------------------------
>
> Key: SPARK-60948
> URL: https://issues.apache.org/jira/browse/SPARK-60948
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect renders a Column as {{Column<'...'>}} of its expression, with
> an operator in the form of a function. The Rust-backed Python client renders
> operators infix, so these PySpark 4.3.0 reprs differ:
> * {{F.col("id") + 1}} is {{Column<'+(id, 1)'>}}; the client gives
> {{Column<'(id + 1)'>}}. The same holds for the other arithmetic and
> comparison operators, for {{&}} and {{|}}, which are {{and(...)}} and
> {{or(...)}}, and for {{between}}, which is {{and(>=(a, 1), <=(a, 2))}}, also
> within {{CASE WHEN}}.
> * {{-F.col("a")}} is {{Column<'negative(a)'>}}, and {{~F.col("a")}} is
> {{Column<'not(a)'>}}; the client gives {{(- a)}} and {{(NOT a)}}.
> * {{F.col("a").cast("int")}} is {{CAST(a AS INT)}}; the client gives {{CAST(a
> AS int)}}.
> * {{F.col("*")}} is {{unresolvedstar()}}; the client gives {{*}}.
> * {{F.col("a")[0]}} is {{a['0']}}; the client gives {{a[0]}}.
> The repr also appears in error messages, such as the one that
> {{SparkSession.sql}} raises for a keyword argument that is not a plain column
> reference. Render Column reprs as PySpark Connect does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
