[
https://issues.apache.org/jira/browse/SPARK-61024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61024:
---------------------------------
Description:
In PySpark, DataFrame methods check their arguments on the client. They raise a
PySparkTypeError of {{NOT_EXPECTED_TYPE}}, with the name, the expected type and
the type of the argument, for: {{select}}, {{drop}}, {{repartition}} and
{{groupBy}} of a value that is neither a Column nor a string (nor an ordinal
for {{groupBy}}); {{sort}} and {{orderBy}} of one that is not a Column, an int
or a string; {{toDF}} of a non-string; {{sample}} of arguments that are not a
bool, a float or an int; {{hint}} of a name that is not a string;
{{dropDuplicates}} and {{dropDuplicatesWithinWatermark}} of a subset that is
not a list or tuple of strings; {{withColumn}} of a value that is not a Column;
and {{withColumns}} and {{withColumnsRenamed}} of a value that is not a dict.
{{hint}} raises {{INVALID_ITEM_FOR_CONTAINER}} for a parameter that is not a
string, a float, an int, a Column or a list of them. {{repartition}},
{{repartitionByRange}} and {{coalesce}} raise {{VALUE_NOT_POSITIVE}} for a
number of partitions that is not positive. {{sort}}, {{repartitionByRange}} and
{{agg}} raise {{CANNOT_BE_EMPTY}} without columns.
The Rust-backed Python client raises a TypeError for the types, such as
{{argument 'col2': 'int' object is not an instance of 'str'}}, whose
getCondition() is not NOT_EXPECTED_TYPE. It accepts the parameters of {{hint}}
of other types, sends the numbers of partitions that are not positive to the
server, which fails, and sorts by, repartitions by range by, and aggregates no
columns. Raise the errors as PySpark does.
was:
In PySpark, DataFrame methods check the types of their arguments on the client,
and raise a PySparkTypeError of {{NOT_EXPECTED_TYPE}} with the name, the
expected type and the type of the argument. Examples: {{select}}, {{drop}} and
{{repartition}} of a value that is neither a Column nor a string, {{toDF}} of a
non-string, {{sort}} of an {{ascending}} that is not a bool, an int or a list,
{{sample}} of a fraction that is not a float, {{hint}} of a name that is not a
string, {{dropDuplicates}} of a string subset, and {{withColumns}} and
{{withColumnsRenamed}} of a value that is not a dict.
The Rust-backed Python client raises a TypeError for these, such as {{argument
'col2': 'int' object is not an instance of 'str'}}, whose getCondition() is not
NOT_EXPECTED_TYPE. Raise them as PySpark does.
Summary: Raise the errors of PySpark for the arguments of DataFrame
methods in the Python client (was: Raise the NOT_EXPECTED_TYPE errors of
PySpark for the arguments of DataFrame methods in the Python client)
> Raise the errors of PySpark for the arguments of DataFrame methods in the
> Python client
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61024
> URL: https://issues.apache.org/jira/browse/SPARK-61024
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, DataFrame methods check their arguments on the client. They raise
> a PySparkTypeError of {{NOT_EXPECTED_TYPE}}, with the name, the expected type
> and the type of the argument, for: {{select}}, {{drop}}, {{repartition}} and
> {{groupBy}} of a value that is neither a Column nor a string (nor an ordinal
> for {{groupBy}}); {{sort}} and {{orderBy}} of one that is not a Column, an
> int or a string; {{toDF}} of a non-string; {{sample}} of arguments that are
> not a bool, a float or an int; {{hint}} of a name that is not a string;
> {{dropDuplicates}} and {{dropDuplicatesWithinWatermark}} of a subset that is
> not a list or tuple of strings; {{withColumn}} of a value that is not a
> Column; and {{withColumns}} and {{withColumnsRenamed}} of a value that is not
> a dict. {{hint}} raises {{INVALID_ITEM_FOR_CONTAINER}} for a parameter that
> is not a string, a float, an int, a Column or a list of them.
> {{repartition}}, {{repartitionByRange}} and {{coalesce}} raise
> {{VALUE_NOT_POSITIVE}} for a number of partitions that is not positive.
> {{sort}}, {{repartitionByRange}} and {{agg}} raise {{CANNOT_BE_EMPTY}}
> without columns.
> The Rust-backed Python client raises a TypeError for the types, such as
> {{argument 'col2': 'int' object is not an instance of 'str'}}, whose
> getCondition() is not NOT_EXPECTED_TYPE. It accepts the parameters of
> {{hint}} of other types, sends the numbers of partitions that are not
> positive to the server, which fails, and sorts by, repartitions by range by,
> and aggregates no columns. Raise the errors as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
