laserninja opened a new pull request, #58788:
URL: https://github.com/apache/spark/pull/58788
### What changes were proposed in this pull request?
Return zero for the infinity norm of a positive-dimensional sparse vector
with no stored entries. Apply the same fix to pyspark.ml.linalg and
pyspark.mllib.linalg, with tests for both APIs.
### Why are the changes needed?
SparseVector.norm passes its empty values array to numpy.linalg.norm, which
raises on the maximum reduction. Dense vectors and sparse vectors storing an
explicit zero return 0.0 for the same logical vector.
```python
from pyspark.ml.linalg import SparseVector
SparseVector(3, [], []).norm(float("inf"))
# ValueError: zero-size array to reduction operation maximum which has no
identity
```
### Does this PR introduce _any_ user-facing change?
Yes. All-zero sparse vectors return 0.0 for infinity norm. Zero-dimensional
vectors and invalid norm orders retain their existing behavior.
### How was this patch tested?
The new regression tests failed before the fix and passed afterward on
Python 3.10.11 with NumPy 2.2.6. The focused run passed 4 tests, including
parameterized subtests:
```bash
PYTHONPATH=python TERM=dumb python -m unittest
pyspark.ml.tests.test_linalg.SparseVectorNormTests
pyspark.mllib.tests.test_linalg.SparseVectorNormTests -q
PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error
--ruff
```
Run the commands with the Python virtual environment activated; the local
run shared an environment from a sibling checkout. The tests call public Python
APIs directly and require no SparkSession. Python compilation, custom-error
checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run
because these fixes are confined to Python-side logic.
Draft pending creation and linking of the corresponding ASF Jira issue.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
