Hyukjin Kwon created SPARK-61542:
------------------------------------
Summary: Convert the counts of limit, offset, tail, head, take,
repartition, repartitionByRange, coalesce and repartitionById as protobuf's
default backend does, refusing a bool, as PySpark does
Key: SPARK-61542
URL: https://issues.apache.org/jira/browse/SPARK-61542
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Under protobuf's default backend (upb) on protobuf 7.34+, which a fresh install
of pyspark-client 4.3.0 gets, official PySpark converts a count that it sets in
a proto int field as upb does: a bool raises TypeError "Field
spark.connect.Limit.limit: Expected an int, got a boolean.", a value that is
not an int raises the TypeError of operator.index, and an int out of the range
of the field raises ValueError "Value out of range: N".
The Python client took these counts through PyO3: a bool was sent as 1 or 0,
and the other errors were PyO3's, prefixed with the name of the argument
(OverflowError for a big int). This applies to DataFrame.limit, offset, tail,
head, take, repartition, repartitionByRange, coalesce and repartitionById.
Each now runs PySpark's own checks first, in PySpark's order (the type checks,
`not numPartitions > 0` with VALUE_NOT_POSITIVE of its str, CANNOT_BE_EMPTY),
and then converts the count as upb does, with the full name of its field. The
helper proto_int is shared with dropna's thresh. A partitionIdCol of
repartitionById that is neither a Column nor a str now raises PySpark's
NOT_EXPECTED_TYPE of `col` (F._to_col) instead of a custom TypeError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]