Hyukjin Kwon created SPARK-61865:
------------------------------------

             Summary: Send and convert the ML params as PySpark does: lists of 
ints of the type of their first element, ints beyond their type, and the copy 
of the validator models
                 Key: SPARK-61865
                 URL: https://issues.apache.org/jira/browse/SPARK-61865
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-61766 and the ML reviews for the ML params of the drop-in 
and the Rust API, against PySpark 4.3.0:
- A list of ints of an ML param (MultilayerPerceptronClassifier.layers, 
VectorSlicer.indices, QuantileDiscretizer.numBucketsArray) is sent as an 
Integer array when every element fits 32 bits, else as a Long array, so a list 
such as [2, 2**32+3, 2] fits a model of layers [2, 3, 2]. PySpark types the 
list by its first element, and its LiteralExpression asserts each element's 
range, so it raises a bare AssertionError at the fit, transform or save. Of 
several unsendable params, PySpark raises the error of the first of its 
_paramMap.
- The Rust API's PredictionModel::num_features truncates a long to an i32, and 
the grid ints beyond a long saturate where PySpark raises VALUE_NOT_BETWEEN.
- The apply_param_map of CrossValidatorModel and TrainValidationSplitModel 
applies the map to the best model only, where PySpark's copy(extra) asserts 
subModels, copies them and sets the model's own params.
- RandomForestClassificationModel::summary() of no summary gives another 
message than PySpark's "No training summary available for this 
RandomForestClassificationModel".
- Docs: the deprecations of StringIndexerModel.labels and ChiSqSelector, and 
the arithmetic of the LR thresholds.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to