Hyukjin Kwon created SPARK-61026:
------------------------------------
Summary: Take the positional fraction and seed of sample, the
ordinals of sort and groupBy, the Columns of drop and the list of selectExpr in
the Python client as PySpark does
Key: SPARK-61026
URL: https://issues.apache.org/jira/browse/SPARK-61026
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark, {{df.sample(withReplacement, fraction, seed)}} takes a fraction and
a seed positionally: {{df.sample(0.5, 42)}} samples a fraction of 0.5 with the
seed 42. {{df.sort}}, {{df.orderBy}} and {{df.sortWithinPartitions}} take
ordinals: {{df.sort(2)}} sorts by the second column and {{df.sort(-2)}} by it
descending, and {{df.groupBy(2)}} groups by the second column. {{df.drop}}
takes Columns as well as column names, such as {{df.drop(df.x)}}, and
{{df.selectExpr}} takes a list of expressions.
The Rust-backed Python client takes the 42 of {{df.sample(0.5, 42)}} as the
fraction, which fails on the server, raises a TypeError for the ordinals of
{{sort}}, {{orderBy}}, {{sortWithinPartitions}} and {{groupBy}}, for the
Columns of {{drop}} and for the list of {{selectExpr}}. Take them as PySpark
does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
