[ 
https://issues.apache.org/jira/browse/SPARK-61880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61880:
-----------------------------------
    Labels: pull-request-available  (was: )

> Raise the argument errors of the DataFrame, Column, GroupedData and functions 
> APIs where PySpark raises them, when the plan is built, with PySpark's errors
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61880
>                 URL: https://issues.apache.org/jira/browse/SPARK-61880
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Many DataFrame, Column, GroupedData and functions APIs of the drop-in convert 
> their arguments when they are called, where PySpark 4.3.0 sets them into the 
> plan's protobuf messages when it builds the plan:
> - A str that UTF-8 cannot encode, such as a lone surrogate, fails at the call 
> in F.col, lit, expr, df[name], colRegex, metadataColumn, alias, selectExpr, 
> filter, withColumnRenamed, withColumnsRenamed, withMetadata, hint, drop, 
> dropDuplicates, toDF, withWatermark, unpivot, transpose, observe, groupBy, 
> rollup, cube, groupingSets, pivot, sort, orderBy, sortWithinPartitions, 
> describe, summary, na, crosstab, freqItems, sampleBy, join(on=[...]), 
> repartitionById and the column arguments and keyword names of UDFs, where 
> PySpark fails at the first request with upb's UnicodeEncodeError. Some raise 
> other errors: agg({name: "max"}) AttributeError, join(df, name) 
> AttributeError, repartition(name) NOT_EXPECTED_TYPE, explain and 
> dropna(how=...) UnicodeEncodeError where PySpark raises VALUE_NOT_ALLOWED, 
> cast of such a type name TypeError; spark.addTag of such a str is refused 
> where PySpark accepts it and its later executions fail.
> - Values that PySpark sets only at plan build are converted at the call: 
> DataFrame.alias(5), withColumnRenamed(1, "z"), withColumnsRenamed({"a": 1}), 
> unpivot of an int name, describe(b"ab"), freqItems(["a", 1]), fillna(2**70), 
> dropna(thresh=2**40), na.replace(subset=[1]), join(on=["a", 1]), 
> spark.range(2**70) and tvf.range; bytes, which protobuf's string fields take, 
> are refused.
> - withColumn and withMetadata of a name that is not a str raise TypeError 
> where PySpark raises AssertionError; F.col(5) raises the AttributeError of 
> endswith where PySpark raises TypeError; F.expr(5), filter(5), 
> selectExpr(b"a") and agg({b"a": ...}) raise other errors; a cast to a 
> DataType subclass is sent as VOID where PySpark raises UNSUPPORTED_OPERATION 
> at plan build.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to