Hyukjin Kwon created SPARK-61409:
------------------------------------
Summary: Raise PySpark's errors for a list among the columns of
repartition, a ternary pow and an argument of Connect's _to_col that is not a
column, and warn of a Column key in getItem, in the Python client
Key: SPARK-61409
URL: https://issues.apache.org/jira/browse/SPARK-61409
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Four differences of the Python client from PySpark 4.3.0:
1. repartition(n, [cols]) and repartitionByRange(n, [cols]) unpack a single
list or tuple of columns. PySpark's give the tuple of *cols itself to _to_cols,
so a list among the columns is NOT_EXPECTED_TYPE ("Column or str", cols), and
repartitionByRange(n, []) is that error too, where the client raised
CANNOT_BE_EMPTY.
2. pow(col, 2, 3) and pow(2, col, 3) silently ignore the modulo. PySpark's
Column.__pow__ and __rpow__ take no modulo, so they raise TypeError
("Column.__pow__() takes 2 positional arguments but 3 were given", and Python's
"unsupported operand type(s) for ** or pow()").
3. Column.getItem(<Column>) gives no FutureWarning; PySpark's warns "A column
as 'key' in getItem is deprecated as of Spark 3.0 ...", as the client's
getField already does for its name.
4. pyspark.sql.connect.functions.builtin._to_col is the lenient _to_col of
pyspark.sql.functions, which passes an argument that is not a Column or a str
on as it is. So pyspark.ml.functions.vector_to_array([1.0]) and
array_to_vector(1) build expressions, and from_protobuf(1, ...) and
to_protobuf([1], ...) raise a plain TypeError, where PySpark raises
NOT_EXPECTED_TYPE ("Column or str", col) for all of them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]