Hyukjin Kwon created SPARK-61542:
------------------------------------

             Summary: Convert the counts of limit, offset, tail, head, take, 
repartition, repartitionByRange, coalesce and repartitionById as protobuf's 
default backend does, refusing a bool, as PySpark does
                 Key: SPARK-61542
                 URL: https://issues.apache.org/jira/browse/SPARK-61542
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Under protobuf's default backend (upb) on protobuf 7.34+, which a fresh install 
of pyspark-client 4.3.0 gets, official PySpark converts a count that it sets in 
a proto int field as upb does: a bool raises TypeError "Field 
spark.connect.Limit.limit: Expected an int, got a boolean.", a value that is 
not an int raises the TypeError of operator.index, and an int out of the range 
of the field raises ValueError "Value out of range: N".

The Python client took these counts through PyO3: a bool was sent as 1 or 0, 
and the other errors were PyO3's, prefixed with the name of the argument 
(OverflowError for a big int). This applies to DataFrame.limit, offset, tail, 
head, take, repartition, repartitionByRange, coalesce and repartitionById.

Each now runs PySpark's own checks first, in PySpark's order (the type checks, 
`not numPartitions > 0` with VALUE_NOT_POSITIVE of its str, CANNOT_BE_EMPTY), 
and then converts the count as upb does, with the full name of its field. The 
helper proto_int is shared with dropna's thresh. A partitionIdCol of 
repartitionById that is neither a Column nor a str now raises PySpark's 
NOT_EXPECTED_TYPE of `col` (F._to_col) instead of a custom TypeError.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to