[ 
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruifeng Zheng updated SPARK-61228:
----------------------------------
    Description: 
Add SparseVectorToArray, an internal Catalyst ML expression that expands a 
sparse vector's size, indices, and double values into a dense array of doubles 
or floats.

Require integer indices and double values with containsNull=false for both 
arrays. Fill missing positions with zero and propagate null inputs. Implement 
interpreted and generated evaluation, allocating the requested output type 
directly.

This change adds only the sparse expansion expression and its tests. The 
existing vector_to_array function and its registration remain unchanged.

Pull request: https://github.com/apache/spark/pull/60224


  was:
The current vector_to_array implementation uses Scala UDFs that deserialize 
each input vector. Add a dedicated Catalyst ML expression, VectorToArray, to 
convert underlying vector rows to arrays of doubles or floats in interpreted 
and generated execution.

Accept vector SQL structs and UDTs whose underlying SQL type matches the vector 
schema, including field nullability and containsNull=false for the indices and 
values arrays. Use VectorToArray(input, targetType) directly in ML 
InternalFunctionRegistration for the existing vector_to_array function. The 
expression also uses vector_to_array as its display name.

Cover ML and MLlib vector UDTs, SQL struct inputs, dense/sparse/empty vectors, 
both output types, schema nullability, and the registered function's output 
name.

Pull request: https://github.com/apache/spark/pull/60224


        Summary: [SQL][ML] Add a SparseVectorToArray Catalyst expression  (was: 
[SQL][ML] Add a dedicated vector_to_array Catalyst expression)

> [SQL][ML] Add a SparseVectorToArray Catalyst expression
> -------------------------------------------------------
>
>                 Key: SPARK-61228
>                 URL: https://issues.apache.org/jira/browse/SPARK-61228
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML, SQL
>    Affects Versions: 5.0.0
>            Reporter: Ruifeng Zheng
>            Priority: Major
>              Labels: pull-request-available
>
> Add SparseVectorToArray, an internal Catalyst ML expression that expands a 
> sparse vector's size, indices, and double values into a dense array of 
> doubles or floats.
> Require integer indices and double values with containsNull=false for both 
> arrays. Fill missing positions with zero and propagate null inputs. Implement 
> interpreted and generated evaluation, allocating the requested output type 
> directly.
> This change adds only the sparse expansion expression and its tests. The 
> existing vector_to_array function and its registration remain unchanged.
> Pull request: https://github.com/apache/spark/pull/60224



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to