zhengruifeng opened a new pull request, #58849: URL: https://github.com/apache/spark/pull/58849
### What changes were proposed in this pull request? This PR adds `BLAS.asum(Vector)` and `BLAS.nrm2(Vector)` to both ML and MLlib and uses them for L1 and L2 vector norms. Dense vectors pass their backing arrays to BLAS, while sparse vectors pass only their stored values. The L2 implementation retains a scaled fallback for overflow and underflow because the Java `VectorBLAS` implementation computes `sqrt(sum(x * x))` directly. ### Why are the changes needed? The existing Scala loops do not use the SIMD BLAS implementation available to Spark. They also return infinity or zero for finite L2 inputs whose squares overflow or underflow. On an Intel Xeon Platinum 8375C with JDK 17 and `VectorBLAS`, a local benchmark measured these speedups over the existing Scala loops: | Stored values | L1 | L2 | | ---: | ---: | ---: | | 16 | 1.6x | 1.2x | | 256 | 7.7x | 7.4x | | 4,096 | 7.8x | 8.2x | | 65,536 | 6.1x | 7.2x | Dense and sparse vectors had equivalent results for the same stored-value count. For all-zero dense L2 vectors, the new implementation was 0.6x at 16 values (about 10 ns instead of 7 ns) and 1.9x at 65,536 values. ### Does this PR introduce _any_ user-facing change? Yes. L1 and L2 vector norm computation is faster. L2 norms also remain finite and nonzero for finite inputs where the previous implementation overflowed or underflowed. For example, the L2 norms of `[1e200, -1e200]` and `[1e-200, -1e-200]` now return approximately `1.414213562373095e200` and `1.414213562373095e-200`, respectively, instead of infinity and zero. ### How was this patch tested? Added dense and sparse regression coverage for ordinary L1/L2 norms and for L2 overflow and underflow cases. Ran: ``` build/sbt "mllib-local/testOnly org.apache.spark.ml.linalg.VectorsSuite" build/sbt "mllib/testOnly org.apache.spark.mllib.linalg.VectorsSuite" ``` All 78 tests passed (37 in `mllib-local`, 41 in `mllib`). The performance results above were measured with a temporary Spark `Benchmark` harness that compared the previous loops with `Vectors.norm`; the harness is not included in this PR. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
