Hyukjin Kwon created SPARK-61728:
------------------------------------
Summary: Send a (value, DataType) argument of the native ML calls
as the literal of that type, as PySpark does
Key: SPARK-61728
URL: https://issues.apache.org/jira/browse/SPARK-61728
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
serialize of pyspark.ml.connect.serialize sends a (value, DataType) pair, a
tuple of two whose second item is a DataType, as LiteralExpression(value,
dataType): the literal of that type. Its asserts raise a bare AssertionError of
a value that is not of the type, such as ("x", IntegerType()), and of a type
that no literal has (MapType, StructType, CharType, VarcharType,
YearMonthIntervalType, a UDT), also with a None value.
The native ML classes of Spark Connect Rust raise the UNSUPPORTED_DATA_TYPE of
a tuple for such an argument of their calls (ChiSquareTest.test,
Correlation.corr, PowerIterationClustering.assignClusters, the members of a
model), where official PySpark 4.3.0 sends the literal for the server to reject
or raises its AssertionError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]