Hyukjin Kwon created SPARK-61391:
------------------------------------

             Summary: Convert the params of the native pyspark.ml.connect 
classes as the type converters of PySpark do
                 Key: SPARK-61391
                 URL: https://issues.apache.org/jira/browse/SPARK-61391
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61358, SPARK-61334 and SPARK-61324, found in a post-merge 
review. The native ML classes of the Python client take their params as fixed 
Rust types, which differ from PySpark's Params type converters: an integral 
float such as maxDepth=3.0 is rejected where PySpark converts it to 3, a bool 
is accepted for an int param where PySpark raises TypeError, None is ignored or 
rejected where PySpark stores it and sends a null literal 
(RandomForestClassifier(bootstrap=None) fits with bootstrap true, PySpark's 
with false), an int out of the 32-bit range raises OverflowError at 
construction where PySpark lets the server validate it, and an empty list 
(VectorAssembler inputCols, setThresholds) is sent where PySpark raises 
CANNOT_INFER_ARRAY_ELEMENT_TYPE. The setters also fail with 'Already borrowed' 
while another thread is in fit.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to