[
https://issues.apache.org/jira/browse/SPARK-61015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61015:
------------------------------------
Assignee: Hyukjin Kwon
> Take the arguments of unpivot, melt and replace in the Python client as
> PySpark does
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61015
> URL: https://issues.apache.org/jira/browse/SPARK-61015
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, the {{ids}} and {{values}} of {{df.unpivot(ids, values,
> variableColumnName, valueColumnName)}} and {{df.melt(...)}}, which is the
> same function, are a column name, a Column, or a list or tuple of them, and
> all four arguments are required. {{df.replace(to_replace, value, subset)}}
> takes a {{subset}} of a column name or a list or tuple of them, and sends
> each value as a literal of its type: a bool, a double (an int as well), a
> string, or {{None}} as a null. It raises {{MIXED_TYPE_REPLACEMENT}} for
> replacements of mixed types, {{ARGUMENT_REQUIRED}} without a {{value}} for a
> {{to_replace}} that is not a dict, {{LENGTH_SHOULD_BE_THE_SAME}} for lists of
> different lengths, and {{NOT_EXPECTED_TYPE}} for arguments of other types.
> The Rust-backed Python client takes only lists: {{df.unpivot("id", ["v"],
> "var", "val")}} raises {{TypeError: argument 'ids': Can't extract `str` to
> `Vec`}}, as does {{df.replace("a", "z", "k")}} for its subset. It sends the
> values of {{replace}} as strings, a string that parses as a number as a
> double: {{df.replace("1", "one")}} and {{df.replace({1: "x"})}} fail on the
> server with {{CAST_INVALID_INPUT}}, {{df.replace(True, False)}} replaces
> nothing, {{df.replace({"a": None})}} replaces "a" with the string 'None', and
> {{df.replace("a", None, ["k"])}} replaces nothing. Take the arguments as
> PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
