Hyukjin Kwon created SPARK-61545:
------------------------------------

             Summary: Let the native ML estimators and scalers take the array 
columns of PySpark's pyspark.ml.connect classes, and scale as PySpark's scalers 
do
                 Key: SPARK-61545
                 URL: https://issues.apache.org/jira/browse/SPARK-61545
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


PySpark's pyspark.ml.connect classes compute with numpy and torch on array 
columns. MaxAbsScaler and StandardScaler take an array<double> (or another 
array of numbers) input column and output an array<double>, and StandardScaler 
scales to (x - mean) / std. LogisticRegression takes array features. The native 
classes of the Python client run Spark's JVM estimators, which require ML 
vectors, so PySpark's own docstring examples failed with "Column features must 
be of type ... VectorUDT ... but was actually ... array<double>".

The native estimators and models now take the array columns that PySpark's take:
- A fit, and the transform of its model, send an array of numbers of the 
scaler's inputCol, or of another estimator's featuresCol, as the vector that 
the JVM reads (array_to_vector; decimal arrays cast to array<double> first). 
The transform keeps the array column as it was in its output.
- The models of MaxAbsScaler and StandardScaler output an array<double> 
(vector_to_array), as PySpark's do.
- StandardScaler sends the JVM's withMean, so that it scales to (x - mean) / 
std as PySpark's does.

docs/compatibility.md describes the remaining differences: the JVM's doubles 
for float arrays, and the vector outputs of LogisticRegression.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to