[
https://issues.apache.org/jira/browse/SPARK-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944865#comment-14944865
]
Yizhi Liu commented on SPARK-10757:
-----------------------------------
[~yanboliang] I'm trying to work with this issue. Here is some of my idea:
Take BlockMatrix as an example, the type of 'blocks' in current constructor is
RDD[((Int, Int), Matrix)], while more precisely it should be RDD[((Int, Int),
T<:Matrix)]. This could cause some problem. If we add a Java-friendly
constructor
def this(blocks: JavaRDD[((Integer, Integer), Matrix)],
rowsPerBlock: Int,
colsPerBlock: Int)
and then construct a BlockMatrix with RDD[(Int, Int), DenseMatrix], the
complier will complain a conflict, like:
method constructor BlockMatrix with alternatives:
(blocks: org.apache.spark.api.java.JavaRDD[((Integer, Integer),
org.apache.spark.mllib.linalg.Matrix)],rowsPerBlock: Int,colsPerBlock:
Int)org.apache.spark.mllib.linalg.distributed.BlockMatrix <and>
(blocks: org.apache.spark.rdd.RDD[((Int, Int),
org.apache.spark.mllib.linalg.Matrix)],rowsPerBlock: Int,colsPerBlock:
Int)org.apache.spark.mllib.linalg.distributed.BlockMatrix
cannot be applied to (org.apache.spark.rdd.RDD[((Int, Int),
org.apache.spark.mllib.linalg.DenseMatrix)], Int, Int)
On the other hand, BlockMatrix should hide the underlying implementation of
sub-matrix, no matter it is dense or sparse. Thus, a class level covariant for
Matrix is improper. So I prefer to add a companion object method to construct a
BlockMatrix, like
def from[M <: Matrix : ClassTag](
blocks: JavaRDD[((Integer, Integer), M)],
rowsPerBlock: Int,
colsPerBlock: Int,
nRows: Long,
nCols: Long): BlockMatrix = {
val rdd = blocks.rdd.map { case ((blockRowIndex, blockColIndex), subMatrix)
=>
((blockRowIndex.toInt, blockColIndex.toInt),
subMatrix.asInstanceOf[Matrix]) }
new BlockMatrix(rdd, rowsPerBlock, colsPerBlock, nRows, nCols)
}
Any discussion would be really appreciated. Please let me know if my idea has
any problem. Thank you!
> Java friendly constructor for distributed matrices
> --------------------------------------------------
>
> Key: SPARK-10757
> URL: https://issues.apache.org/jira/browse/SPARK-10757
> Project: Spark
> Issue Type: Improvement
> Components: Java API, MLlib
> Reporter: Yanbo Liang
> Priority: Minor
>
> Currently users can not construct
> BlockMatrix/RowMatrix/IndexedRowMatrix/CoordinateMatrix at Java side because
> that these classes did not provide java friendly constructors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
