[
https://issues.apache.org/jira/browse/SPARK-60981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60981:
-----------------------------------
Labels: pull-request-available (was: )
> Alias a Column with several names, and with metadata of any JSON values, in
> Spark Connect Rust as PySpark does
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60981
> URL: https://issues.apache.org/jira/browse/SPARK-60981
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{Column.alias(*alias, metadata=None)}} names a column with each
> given name, so a generator's columns can all be named:
> {{F.explode(F.col("m")).alias("k", "v")}} names the key and the value of a
> map. The metadata is any JSON-serializable dict, which is sent as
> {{json.dumps(metadata)}}, and metadata with more than one name raises
> {{PySparkValueError}} {{ONLY_ALLOWED_FOR_SINGLE_COLUMN}}. The Rust-backed
> Python client differs:
> * It keeps only the first name, so {{spark.sql("SELECT map('k1', 1) AS
> m").select(F.explode("m").alias("k", "v"))}} fails on the server with
> {{UDTF_ALIAS_NUMBER_MISMATCH}}, where PySpark 4.3.0 gives the columns {{k}}
> and {{v}}.
> * It takes metadata of string values only, so {{F.col("id").alias("x",
> metadata={"k": 1})}} raises {{TypeError: 'int' object is not an instance of
> 'str'}}.
> The Rust {{Column}} has no alias of several names either. Alias with several
> names, and with metadata of any JSON values, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
