[
https://issues.apache.org/jira/browse/SPARK-61024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61024:
---------------------------------
Description:
In PySpark, DataFrame methods check the types of their arguments on the client,
and raise a PySparkTypeError of {{NOT_EXPECTED_TYPE}} with the name, the
expected type and the type of the argument. Examples: {{select}}, {{drop}} and
{{repartition}} of a value that is neither a Column nor a string, {{toDF}} of a
non-string, {{sort}} of an {{ascending}} that is not a bool, an int or a list,
{{sample}} of a fraction that is not a float, {{hint}} of a name that is not a
string, {{dropDuplicates}} of a string subset, and {{withColumns}} and
{{withColumnsRenamed}} of a value that is not a dict.
The Rust-backed Python client raises a TypeError for these, such as {{argument
'col2': 'int' object is not an instance of 'str'}}, whose getCondition() is not
NOT_EXPECTED_TYPE. Raise them as PySpark does.
was:
In PySpark, DataFrame methods check the types of their arguments on the client,
and raise a PySparkTypeError of {{NOT_EXPECTED_TYPE}} with the name, the
expected type and the type of the argument. Examples: {{select}}, {{drop}} and
{{repartition}} of a value that is neither a Column nor a string, {{toDF}} of a
non-string, {{sort}} of an {{ascending}} that is not a bool, an int or a list,
{{sample}} of a fraction that is not a float, {{hint}} of a name that is not a
string, {{dropDuplicates}} of a string subset, {{withColumns}} and
{{withColumnsRenamed}} of a value that is not a dict, {{cov}} and {{crosstab}}
of a column that is not a string, {{freqItems}} of {{cols}} and
{{approxQuantile}} of {{probabilities}} that are not a list or tuple, and
{{summary}} of a statistic that is not a string.
The Rust-backed Python client raises a TypeError for these, such as {{argument
'col2': 'int' object is not an instance of 'str'}}, whose getCondition() is not
NOT_EXPECTED_TYPE. Raise them as PySpark does.
> Raise the NOT_EXPECTED_TYPE errors of PySpark for the arguments of DataFrame
> methods in the Python client
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61024
> URL: https://issues.apache.org/jira/browse/SPARK-61024
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, DataFrame methods check the types of their arguments on the
> client, and raise a PySparkTypeError of {{NOT_EXPECTED_TYPE}} with the name,
> the expected type and the type of the argument. Examples: {{select}},
> {{drop}} and {{repartition}} of a value that is neither a Column nor a
> string, {{toDF}} of a non-string, {{sort}} of an {{ascending}} that is not a
> bool, an int or a list, {{sample}} of a fraction that is not a float,
> {{hint}} of a name that is not a string, {{dropDuplicates}} of a string
> subset, and {{withColumns}} and {{withColumnsRenamed}} of a value that is not
> a dict.
> The Rust-backed Python client raises a TypeError for these, such as
> {{argument 'col2': 'int' object is not an instance of 'str'}}, whose
> getCondition() is not NOT_EXPECTED_TYPE. Raise them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
