[
https://issues.apache.org/jira/browse/SPARK-61728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61728:
-----------------------------------
Labels: pull-request-available (was: )
> Send a (value, DataType) argument of the native ML calls as the literal of
> that type, as PySpark does
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-61728
> URL: https://issues.apache.org/jira/browse/SPARK-61728
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> serialize of pyspark.ml.connect.serialize sends a (value, DataType) pair, a
> tuple of two whose second item is a DataType, as LiteralExpression(value,
> dataType): the literal of that type. Its asserts raise a bare AssertionError
> of a value that is not of the type, such as ("x", IntegerType()), and of a
> type that no literal has (MapType, StructType, CharType, VarcharType,
> YearMonthIntervalType, a UDT), also with a None value.
> The native ML classes of Spark Connect Rust raise the UNSUPPORTED_DATA_TYPE
> of a tuple for such an argument of their calls (ChiSquareTest.test,
> Correlation.corr, PowerIterationClustering.assignClusters, the members of a
> model), where official PySpark 4.3.0 sends the literal for the server to
> reject or raises its AssertionError.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]