Github user AnthonyTruchet commented on the issue: https://github.com/apache/spark/pull/16038 Actually aggregating (and thus sending on the network) on quite dense SparseVectors with 10s of million elements is not to taken lightly. This would required serious benchmarking. What I tell is that we can not do that lightly because we know that aggregating `acc += partial` when `partial` is sparse and `acc` dense is efficient locally (i.e. within a partition) and that after a partition has been aggregated the vector is already quite dense (enough so that the dense and sparse representation have similar size). Anyway if you consider this is not worth an new API and an optimization in core, the discusion should probably go on on the MLlib only PR #16037 for which we already have a specific work around. I just naively assumed that aggregating over big accumulator was typical enough to get some support from core, if this is not the case, I'll fall back on a MLlib only workaround. Feel free to close this PR is you deem it (ir)relevant.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org