Hyukjin Kwon created SPARK-61880:
------------------------------------

             Summary: Raise the argument errors of the DataFrame, Column, 
GroupedData and functions APIs where PySpark raises them, when the plan is 
built, with PySpark's errors
                 Key: SPARK-61880
                 URL: https://issues.apache.org/jira/browse/SPARK-61880
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Many DataFrame, Column, GroupedData and functions APIs of the drop-in convert 
their arguments when they are called, where PySpark 4.3.0 sets them into the 
plan's protobuf messages when it builds the plan:
- A str that UTF-8 cannot encode, such as a lone surrogate, fails at the call 
in F.col, lit, expr, df[name], colRegex, metadataColumn, alias, selectExpr, 
filter, withColumnRenamed, withColumnsRenamed, withMetadata, hint, drop, 
dropDuplicates, toDF, withWatermark, unpivot, transpose, observe, groupBy, 
rollup, cube, groupingSets, pivot, sort, orderBy, sortWithinPartitions, 
describe, summary, na, crosstab, freqItems, sampleBy, join(on=[...]), 
repartitionById and the column arguments and keyword names of UDFs, where 
PySpark fails at the first request with upb's UnicodeEncodeError. Some raise 
other errors: agg({name: "max"}) AttributeError, join(df, name) AttributeError, 
repartition(name) NOT_EXPECTED_TYPE, explain and dropna(how=...) 
UnicodeEncodeError where PySpark raises VALUE_NOT_ALLOWED, cast of such a type 
name TypeError; spark.addTag of such a str is refused where PySpark accepts it 
and its later executions fail.
- Values that PySpark sets only at plan build are converted at the call: 
DataFrame.alias(5), withColumnRenamed(1, "z"), withColumnsRenamed({"a": 1}), 
unpivot of an int name, describe(b"ab"), freqItems(["a", 1]), fillna(2**70), 
dropna(thresh=2**40), na.replace(subset=[1]), join(on=["a", 1]), 
spark.range(2**70) and tvf.range; bytes, which protobuf's string fields take, 
are refused.
- withColumn and withMetadata of a name that is not a str raise TypeError where 
PySpark raises AssertionError; F.col(5) raises the AttributeError of endswith 
where PySpark raises TypeError; F.expr(5), filter(5), selectExpr(b"a") and 
agg({b"a": ...}) raise other errors; a cast to a DataType subclass is sent as 
VOID where PySpark raises UNSUPPORTED_OPERATION at plan build.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to