Ruifeng Zheng created SPARK-61518:
-------------------------------------

             Summary: Replace vector_to_array UDFs with a runtime-replaceable 
expression
                 Key: SPARK-61518
                 URL: https://issues.apache.org/jira/browse/SPARK-61518
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


vector_to_array currently deserializes both ML vector UDT versions into JVM 
objects and executes a Scala UDF for each output element type.

Replace the conversion UDFs with a runtime-replaceable VectorToArray Catalyst 
expression over the canonical vector struct. The internal ML function builder 
unwraps the UDT with UnwrapUDT. The replacement casts the dense values array to 
the requested double/float array type or expands a sparse vector with the 
existing SparseVectorToArray expression. Preserve vector-schema checks, 
including non-null indices/values elements.

Null vectors propagate null rather than throwing from the UDF, and incompatible 
inputs fail analysis. Both ML and legacy MLlib vector UDTs and the 
float64/float32 dtype options remain supported.

Add Catalyst evaluation, schema, and UDT tests, and update the ML integration 
coverage to verify nulls, empty and special values, both UDTs and dtypes, and 
the absence of conversion UDFs in the optimized plan.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to