zhengruifeng opened a new pull request, #57967:
URL: https://github.com/apache/spark/pull/57967
### What changes were proposed in this pull request?
Update the DataFrame Summarizer aggregate to consume vectors from their
internal SQL
representation. A nonzero iterator reads dense values and sparse index-value
pairs directly from
Catalyst ArrayData. Count-only summaries avoid iterating over vector values.
### Why are the changes needed?
Summarizer currently calls VectorUDT.deserialize for every input row. This
copies the dense values
array, or both sparse indices and values arrays, and then allocates a Vector
wrapper before
aggregation. Reading the internal row directly avoids these per-row copies
and Vector allocation.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The existing SummarizerSuite covers dense, sparse, weighted, count-only, and
multi-metric summary
behavior.
build/sbt 'mllib/testOnly org.apache.spark.ml.stat.SummarizerSuite'
build/sbt mllib/scalastyle
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
