bearomorphism opened a new pull request, #59585:
URL: https://github.com/apache/spark/pull/59585
### What changes were proposed in this pull request?
- Make `pyspark.ml.linalg.Vector` and `pyspark.mllib.linalg.Vector` abstract
base classes using `ABCMeta` and `@abstractmethod`.
- Declare the shared `norm()` and `squared_distance()` contracts alongside
`toArray()` and `__len__()`; the legacy MLlib base also declares `asML()`
abstract.
- Align concrete norm return annotations with `np.float64` and the ML
dense-vector distance input with `VectorLike`.
- Remove the eight type suppressions made unnecessary by those declarations
and signature changes, including the distance call in `KMeansModel.predict()`.
- Place the class docstrings before `__UDT__` so they are recognized as
docstrings. Preserve the UDT attributes and concrete vector implementations.
Changes to `NormType`/`VectorLike` aliases, broader NumPy typing, arithmetic
delegation, factories, iteration, and parsing are explicitly out of scope.
### Why are the changes needed?
Both Vector bases describe themselves as abstract but currently permit
direct instantiation and defer failure until an unimplemented method is called.
Their incomplete shared interfaces also require callers to suppress
missing-member errors for operations supported by both concrete vector classes.
An explicit abstract contract enforces implementation at construction time
and lets type checkers resolve the shared operations without those suppressions.
### Does this PR introduce _any_ user-facing change?
Yes. Direct construction of either `Vector` base, or of a subclass missing a
required abstract method, now raises `TypeError` at instantiation rather than
allowing construction and potentially raising `NotImplementedError` later.
Required methods are `norm()`, `squared_distance()`, `toArray()`, and
`__len__()`, plus `asML()` for legacy MLlib vectors. Downstream custom
subclasses must implement these methods. Introducing `ABCMeta` may also require
compatibility review for downstream custom metaclasses.
Built-in `DenseVector` and `SparseVector` already implement these methods.
Their numerical implementations and serialization formats are unchanged. Static
annotations for norm results and the ML dense-vector distance input are more
precise.
### How was this patch tested?
This draft adds no committed test cases; regression-test additions are
deferred. Validation used existing tests and temporary contract checks.
- **37 existing tests passed**, with no failures, errors, or skips: both
`VectorTests` classes and both UDT serialization/schema-inference selections.
```bash
source .venv/bin/activate
SPARK_LOCAL_IP=localhost python/run-tests --keep-test-output --parallelism
2 \
--testnames 'pyspark.ml.tests.test_linalg VectorTests
VectorUDTTests.test_serialization
VectorUDTTests.test_infer_schema,pyspark.mllib.tests.test_linalg VectorTests
VectorUDTTests.test_serialization VectorUDTTests.test_infer_schema'
```
- Python-only smoke checks passed for abstract/incomplete-subclass
rejection, complete subclasses, concrete dense/sparse vectors, and
`KMeansModel([[0, 0], [10, 10]]).predict([1, 1]) == 0`.
- Focused mypy checks passed for the three changed files and temporary
`assert_type` checks for operations through both abstract Vector types.
```bash
mypy --config-file python/mypy.ini --follow-imports=silent \
python/pyspark/ml/linalg/__init__.py \
python/pyspark/mllib/linalg/__init__.py \
python/pyspark/mllib/clustering.py
```
- Ruff lint, Ruff formatting checks, and `git diff --check` passed.
- **Repository-wide typing is not green in this environment:** both the
patch and the unmodified HEAD baseline report the same 22 mypy errors in nine
unrelated pandas-related files. The comparison used `--shadow-file` for all
three modified sources and found no introduced diagnostics. All 11 existing ML
typing fixtures fail on unexpected diagnostics drawn from that same established
baseline set. These unrelated failures are not modified in this draft.
Validation used Python 3.13.3 on arm64 with the repository's development
dependency group and existing Spark JVM artifacts.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 1.3.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
