Github user johnc1231 commented on the issue:
https://github.com/apache/spark/pull/17459
@viirya I think we definitely care about giving users the ability to make
either dense or sparse Block matrices. I made a 100k by 10k IndexedRowMatrix of
random doubles, then converted it to a BlockMatrix to multiply it by its
transpose. With the current toBlockMatrix implementation, that took 252 seconds
on 128 cores. With my implementation, that took 35 seconds. The backing of a
BlockMatrix matters a lot, and we need to let users be explicit about it.
I considered having toBlockMatrix check if the rows of IndexedRowMatrix
were dense or sparse, but there is no guarantee of consistency. Like, an
IndexedRowMatrix could be a mix of Dense and Sparse Vectors. In that case, it
would not be clear what type of BlockMatrix to create. A decent approximation
of this would be to just decide the matrix type based on the first vector we
look at in the iterator we get from groupByKey, but I still think it's best to
be explicit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
