[
https://issues.apache.org/jira/browse/SPARK-61015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61015:
---------------------------------
Description:
In PySpark, the {{ids}} and {{values}} of {{df.unpivot(ids, values,
variableColumnName, valueColumnName)}} and {{df.melt(...)}}, which is the same
function, are a column name, a Column, or a list or tuple of them.
{{df.replace(to_replace, value, subset)}} takes a {{subset}} of a column name
or a list or tuple of them, and sends each value as a literal of its type,
{{None}} as null. It raises {{MIXED_TYPE_REPLACEMENT}} for replacements of
mixed types, {{ARGUMENT_REQUIRED}} without a {{value}} for a non-dict
{{to_replace}}, {{LENGTH_SHOULD_BE_THE_SAME}} for lists of different lengths,
and the {{NOT_...}} errors of the types of its arguments.
The Rust-backed Python client takes only lists: {{df.unpivot("id", ["v"],
"var", "val")}} raises {{TypeError: argument 'ids': Can't extract `str` to
`Vec`}}, as does {{df.replace("a", "z", "k")}} for its subset. It sends the
values of {{replace}} as strings, so {{df.replace("a", None, ["k"])}} replaces
with the string {{'None'}}, and a mixed replacement fails on the server with
{{CAST_INVALID_INPUT}}. Take the arguments as PySpark does.
was:
In PySpark, the {{ids}} and {{values}} of {{df.unpivot(ids, values,
variableColumnName, valueColumnName)}} and {{df.melt(...)}} are a column name,
a Column, or a list or tuple of them, and the {{subset}} of
{{df.replace(to_replace, value, subset)}} is a column name or a list or tuple
of them.
The Rust-backed Python client takes only lists: {{df.unpivot("id", ["v"],
"var", "val")}} raises {{TypeError: argument 'ids': Can't extract `str` to
`Vec`}}, as does {{df.replace("a", "z", "k")}} for its subset. Take them as
PySpark does.
Summary: Take the arguments of unpivot, melt and replace in the Python
client as PySpark does (was: Take a column name or a Column for the ids and
values of unpivot and melt, and the subset of replace, in the Python client as
PySpark does)
> Take the arguments of unpivot, melt and replace in the Python client as
> PySpark does
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61015
> URL: https://issues.apache.org/jira/browse/SPARK-61015
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, the {{ids}} and {{values}} of {{df.unpivot(ids, values,
> variableColumnName, valueColumnName)}} and {{df.melt(...)}}, which is the
> same function, are a column name, a Column, or a list or tuple of them.
> {{df.replace(to_replace, value, subset)}} takes a {{subset}} of a column name
> or a list or tuple of them, and sends each value as a literal of its type,
> {{None}} as null. It raises {{MIXED_TYPE_REPLACEMENT}} for replacements of
> mixed types, {{ARGUMENT_REQUIRED}} without a {{value}} for a non-dict
> {{to_replace}}, {{LENGTH_SHOULD_BE_THE_SAME}} for lists of different lengths,
> and the {{NOT_...}} errors of the types of its arguments.
> The Rust-backed Python client takes only lists: {{df.unpivot("id", ["v"],
> "var", "val")}} raises {{TypeError: argument 'ids': Can't extract `str` to
> `Vec`}}, as does {{df.replace("a", "z", "k")}} for its subset. It sends the
> values of {{replace}} as strings, so {{df.replace("a", None, ["k"])}}
> replaces with the string {{'None'}}, and a mixed replacement fails on the
> server with {{CAST_INVALID_INPUT}}. Take the arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
