[
https://issues.apache.org/jira/browse/SPARK-61409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61409:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise PySpark's errors for a list among the columns of repartition, a ternary
> pow and an argument of Connect's _to_col that is not a column, and warn of a
> Column key in getItem, in the Python client
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61409
> URL: https://issues.apache.org/jira/browse/SPARK-61409
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Four differences of the Python client from PySpark 4.3.0:
> 1. repartition(n, [cols]) and repartitionByRange(n, [cols]) unpack a single
> list or tuple of columns. PySpark's give the tuple of *cols itself to
> _to_cols, so a list among the columns is NOT_EXPECTED_TYPE ("Column or str",
> cols), and repartitionByRange(n, []) is that error too, where the client
> raised CANNOT_BE_EMPTY.
> 2. pow(col, 2, 3) and pow(2, col, 3) silently ignore the modulo. PySpark's
> Column.__pow__ and __rpow__ take no modulo, so they raise TypeError
> ("Column.__pow__() takes 2 positional arguments but 3 were given", and
> Python's "unsupported operand type(s) for ** or pow()").
> 3. Column.getItem(<Column>) gives no FutureWarning; PySpark's warns "A column
> as 'key' in getItem is deprecated as of Spark 3.0 ...", as the client's
> getField already does for its name.
> 4. pyspark.sql.connect.functions.builtin._to_col is the lenient _to_col of
> pyspark.sql.functions, which passes an argument that is not a Column or a str
> on as it is. So pyspark.ml.functions.vector_to_array([1.0]) and
> array_to_vector(1) build expressions, and from_protobuf(1, ...) and
> to_protobuf([1], ...) raise a plain TypeError, where PySpark raises
> NOT_EXPECTED_TYPE ("Column or str", col) for all of them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]