[
https://issues.apache.org/jira/browse/SPARK-61228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61228:
----------------------------------
Description:
Add SparseVectorToArray, an internal Catalyst ML expression that accepts the
standard four-field SQL struct representation of a sparse MLlib vector and
expands its size, indices, and values into a dense array of doubles or floats.
Require the vector SQL schema, including field nullability and
containsNull=false for the indices and values arrays. Fill missing positions
with zero and propagate null structs or null size, indices, or values fields.
Implement interpreted and generated evaluation, allocating the requested output
type directly.
The struct input allows composition with UnwrapUDT. Test this composition along
with schema nullability and conversion behavior. This change adds only the
sparse expansion expression and its tests; existing vector_to_array behavior
and registration remain unchanged.
Pull request: https://github.com/apache/spark/pull/60224
was:
Add SparseVectorToArray, an internal Catalyst ML expression that expands a
sparse vector's size, indices, and double values into a dense array of doubles
or floats.
Require integer indices and double values with containsNull=false for both
arrays. Fill missing positions with zero and propagate null inputs. Implement
interpreted and generated evaluation, allocating the requested output type
directly.
This change adds only the sparse expansion expression and its tests. The
existing vector_to_array function and its registration remain unchanged.
Pull request: https://github.com/apache/spark/pull/60224
> [SQL][ML] Add a SparseVectorToArray Catalyst expression
> -------------------------------------------------------
>
> Key: SPARK-61228
> URL: https://issues.apache.org/jira/browse/SPARK-61228
> Project: Spark
> Issue Type: Improvement
> Components: ML, SQL
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> Add SparseVectorToArray, an internal Catalyst ML expression that accepts the
> standard four-field SQL struct representation of a sparse MLlib vector and
> expands its size, indices, and values into a dense array of doubles or floats.
> Require the vector SQL schema, including field nullability and
> containsNull=false for the indices and values arrays. Fill missing positions
> with zero and propagate null structs or null size, indices, or values fields.
> Implement interpreted and generated evaluation, allocating the requested
> output type directly.
> The struct input allows composition with UnwrapUDT. Test this composition
> along with schema nullability and conversion behavior. This change adds only
> the sparse expansion expression and its tests; existing vector_to_array
> behavior and registration remain unchanged.
> Pull request: https://github.com/apache/spark/pull/60224
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]