zhengruifeng opened a new pull request, #58822:
URL: https://github.com/apache/spark/pull/58822
### What changes were proposed in this pull request?
This change adds a package-private `RDD.treeAggregateToRDD` helper that
keeps a tree aggregate as a
single-partition RDD. `RowMatrix` uses this helper to expand Gramian and
covariance matrices on an
executor. For PCA based on a local covariance matrix, the Breeze SVD also
runs in that final
executor task, and only the principal components and explained variance are
returned to the driver.
### Why are the changes needed?
The existing tree aggregation can combine its final values on an executor,
but it always returns
the aggregate to the driver. PCA then expands the packed covariance matrix
and performs the local
SVD on the driver. Consequently, the driver must hold the packed aggregate,
full covariance
matrix, and SVD workspace even though the PCA result is smaller when `k` is
less than the number of
features.
Keeping the aggregate as an RDD allows the local matrix finalization to run
on an executor and
reduces driver memory and CPU pressure.
### Does this PR introduce _any_ user-facing change?
No. This changes the execution location of internal RowMatrix computations
without changing the
public APIs or their numerical results.
### How was this patch tested?
The following test compilation and suites passed:
```
build/sbt core/Test/compile mllib/Test/compile
build/sbt 'core/testOnly org.apache.spark.rdd.RDDSuite' \
'mllib/testOnly org.apache.spark.mllib.linalg.distributed.RowMatrixSuite' \
'mllib/testOnly org.apache.spark.mllib.feature.PCASuite'
./dev/lint-scala
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
