SEPURI-SAI-KRISHNA opened a new pull request, #58149:
URL: https://github.com/apache/spark/pull/58149
PR TITLE:
[SPARK-58897][SQL] Propagate NaN through the infinity norm in vector_norm
--------------------------------------------------------------------------
### What changes were proposed in this pull request?
This PR makes the infinity-norm branch of `vector_norm` propagate `NaN`
elements instead of
silently skipping them.
`VectorFunctionImplUtils.vectorInfNorm` computed the maximum absolute value
with a hand-rolled
comparison against a running maximum seeded at zero:
```java
float maxAbs = 0.0f;
...
float absVal = Math.abs(vec.getFloat(i));
if (absVal > maxAbs) {
maxAbs = absVal;
}
```
Every comparison involving `NaN` is false under IEEE 754, so a `NaN` element
never became the
running maximum and never reached the result. The loop now uses `Math.max`,
which returns `NaN`
when either argument is `NaN`. The accumulator is widened to `double` to
match `vectorL1Norm`
and `vectorL2Norm` and the method's own "in double precision" contract; the
infinity norm of a
float vector cannot overflow the float range, so this part is for
consistency rather than a
second fix.
No change is needed in `vectorNormalize`: `vectorNormalizeWithNorm` guards
only `norm == 0.0d`,
so a `NaN` norm falls through to the division and yields an all-`NaN`
vector, which is what
degree 2.0 already returns.
`VectorNorm` is `RuntimeReplaceable` and lowers to a single `StaticInvoke` on
`VectorFunctionImplUtils`, so the interpreted and codegen paths share this
one implementation.
### Why are the changes needed?
`vector_norm` with degree infinity returns a wrong result for any vector
containing `NaN`. The
single-element case is the damaging one: a vector of `NaN` is reported as
having infinity norm
`0.0`, making it indistinguishable from the zero vector, which in turn makes
`vector_normalize`
return `NULL` through its zero-norm path.
```sql
SELECT vector_norm(array(float('nan')), float('inf')); --
0.0 expected NaN
SELECT vector_norm(array(float('nan'), 5.0F), float('inf')); --
5.0 expected NaN
SELECT vector_norm(array(float('nan'), float('inf')), float('inf')); --
Infinity expected NaN
SELECT vector_normalize(array(float('nan')), float('inf')); --
NULL expected [NaN]
```
The infinity norm is the only degree that behaves this way; degrees 1.0 and
2.0 both propagate
`NaN`. It is also the only max-like function in Spark that does not treat
`NaN` as the largest
value:
```sql
SELECT greatest(float('nan'), 5.0F); -- NaN
SELECT array_max(array(float('nan'), 5.0F)); -- NaN
SELECT max(v) FROM values (float('nan')), (5.0F) AS t(v); -- NaN
SELECT sort_array(array(float('nan'), 5.0F)); -- [5.0, NaN]
```
Since the infinity norm is defined as the maximum absolute value, it should
follow the same
convention as `max` and `array_max`.
### Does this PR introduce _any_ user-facing change?
Yes. `vector_norm(v, float('inf'))` now returns `NaN` when `v` contains a
`NaN` element, and
`vector_normalize(v, float('inf'))` returns an all-`NaN` vector rather than
`NULL`. Both are bug
fixes relative to released 4.2.0.
| Query | Before | After |
| --- | --- | --- |
| `vector_norm(array(float('nan')), float('inf'))` | `0.0` | `NaN` |
| `vector_norm(array(float('nan'), 5.0F), float('inf'))` | `5.0` | `NaN` |
| `vector_norm(array(float('nan'), float('inf')), float('inf'))` |
`Infinity` | `NaN` |
| `vector_normalize(array(float('nan')), float('inf'))` | `NULL` | `[NaN]` |
| `vector_normalize(array(float('nan'), 5.0F), float('inf'))` | `[NaN, 0.0]`
| `[NaN, NaN]` |
Behaviour for empty vectors (`0.0`), vectors containing `NULL` (`NULL`), and
genuinely zero
vectors is unchanged. A `NULL` element continues to take precedence over a
`NaN` element.
### How was this patch tested?
Added a `SPARK-58897` section to `sql-tests/inputs/vector-norm.sql` covering
the single-element
`NaN` case, `NaN` in either position, `NaN` alongside an infinite element,
`NaN` inside the
16-element unrolled loop, the `NULL`-takes-precedence case, the L1 and L2
degrees as regression
guards, and both `vector_normalize` cases. The golden file was regenerated;
the resulting diff
is purely additive, with no existing expectation changed.
```
build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z vector"
build/sbt "sql/testOnly org.apache.spark.sql.MiscFunctionsSuite -- -z
\"vector functions\""
```
All pass, including the existing `vector-distance.sql` and `vector-agg.sql`
golden files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
