Hyukjin Kwon created SPARK-61880:
------------------------------------
Summary: Raise the argument errors of the DataFrame, Column,
GroupedData and functions APIs where PySpark raises them, when the plan is
built, with PySpark's errors
Key: SPARK-61880
URL: https://issues.apache.org/jira/browse/SPARK-61880
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Many DataFrame, Column, GroupedData and functions APIs of the drop-in convert
their arguments when they are called, where PySpark 4.3.0 sets them into the
plan's protobuf messages when it builds the plan:
- A str that UTF-8 cannot encode, such as a lone surrogate, fails at the call
in F.col, lit, expr, df[name], colRegex, metadataColumn, alias, selectExpr,
filter, withColumnRenamed, withColumnsRenamed, withMetadata, hint, drop,
dropDuplicates, toDF, withWatermark, unpivot, transpose, observe, groupBy,
rollup, cube, groupingSets, pivot, sort, orderBy, sortWithinPartitions,
describe, summary, na, crosstab, freqItems, sampleBy, join(on=[...]),
repartitionById and the column arguments and keyword names of UDFs, where
PySpark fails at the first request with upb's UnicodeEncodeError. Some raise
other errors: agg({name: "max"}) AttributeError, join(df, name) AttributeError,
repartition(name) NOT_EXPECTED_TYPE, explain and dropna(how=...)
UnicodeEncodeError where PySpark raises VALUE_NOT_ALLOWED, cast of such a type
name TypeError; spark.addTag of such a str is refused where PySpark accepts it
and its later executions fail.
- Values that PySpark sets only at plan build are converted at the call:
DataFrame.alias(5), withColumnRenamed(1, "z"), withColumnsRenamed({"a": 1}),
unpivot of an int name, describe(b"ab"), freqItems(["a", 1]), fillna(2**70),
dropna(thresh=2**40), na.replace(subset=[1]), join(on=["a", 1]),
spark.range(2**70) and tvf.range; bytes, which protobuf's string fields take,
are refused.
- withColumn and withMetadata of a name that is not a str raise TypeError where
PySpark raises AssertionError; F.col(5) raises the AttributeError of endswith
where PySpark raises TypeError; F.expr(5), filter(5), selectExpr(b"a") and
agg({b"a": ...}) raise other errors; a cast to a DataType subclass is sent as
VOID where PySpark raises UNSUPPORTED_OPERATION at plan build.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]