[
https://issues.apache.org/jira/browse/SPARK-61865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61865:
------------------------------------
Assignee: Hyukjin Kwon
> Send and convert the ML params as PySpark does: lists of ints of the type of
> their first element, ints beyond their type, and the copy of the validator
> models
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61865
> URL: https://issues.apache.org/jira/browse/SPARK-61865
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61766 and the ML reviews for the ML params of the drop-in
> and the Rust API, against PySpark 4.3.0:
> - A list of ints of an ML param (MultilayerPerceptronClassifier.layers,
> VectorSlicer.indices, QuantileDiscretizer.numBucketsArray) is sent as an
> Integer array when every element fits 32 bits, else as a Long array, so a
> list such as [2, 2**32+3, 2] fits a model of layers [2, 3, 2]. PySpark types
> the list by its first element, and its LiteralExpression asserts each
> element's range, so it raises a bare AssertionError at the fit, transform or
> save. Of several unsendable params, PySpark raises the error of the first of
> its _paramMap.
> - The Rust API's PredictionModel::num_features truncates a long to an i32,
> and the grid ints beyond a long saturate where PySpark raises
> VALUE_NOT_BETWEEN.
> - The apply_param_map of CrossValidatorModel and TrainValidationSplitModel
> applies the map to the best model only, where PySpark's copy(extra) asserts
> subModels, copies them and sets the model's own params.
> - RandomForestClassificationModel::summary() of no summary gives another
> message than PySpark's "No training summary available for this
> RandomForestClassificationModel".
> - Docs: the deprecations of StringIndexerModel.labels and ChiSqSelector, and
> the arithmetic of the LR thresholds.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]