Ruifeng Zheng created SPARK-61364:
-------------------------------------
Summary: Check vector schema nullability in VectorDotProduct
Key: SPARK-61364
URL: https://issues.apache.org/jira/browse/SPARK-61364
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
VectorDotProduct currently uses StructType directly with ExpectsInputTypes,
which accepts any struct without validating the vector schema or its nested
nullability. In particular, nullable indices or values elements can reach code
that assumes primitive array elements are non-null.
Validate both operands against the vector SQL schema with compatible
nullability: require the expected field names and types and non-null
indices/values elements, while allowing stricter non-nullable struct fields.
Preserve the vector_dot_product(column, Vector) overload by constructing its
primitive arrays with typed literals.
Add Catalyst coverage for both operands, malformed schemas, nullable array
elements, and compatible stricter fields. Run the existing ML
vector_dot_product test for dense and sparse constant vectors.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]