[ https://issues.apache.org/jira/browse/MAHOUT-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941541#comment-13941541 ]
Dmitriy Lyubimov commented on MAHOUT-1464: ------------------------------------------ Oh, you mean in case of sparse row vectors. You are probably right. indeed, there's currently a SparseMatrix there in this case. I think it should be SparseRowMatrix of course. most of the cases should benefit from it. Problem is, like i said, mapblock doesn't really form it; nor any other physical operator has any knowledge what formed it. It is possible to optimize the entire operator fusion chain based on subsequent operator preferred type, that's actually a very neat idea for in-core speed optimization; but i have no capacity to pursue this technique at the moment. It needs some digestion anyway (at least on my end). It requires experiments with in-core operations. At the first glance, most non-multiplicative operators would be ok with row-wise matrix, as well as deblockifying views. > RowSimilarityJob on Spark > ------------------------- > > Key: MAHOUT-1464 > URL: https://issues.apache.org/jira/browse/MAHOUT-1464 > Project: Mahout > Issue Type: Improvement > Components: Collaborative Filtering > Affects Versions: 0.9 > Environment: hadoop, spark > Reporter: Pat Ferrel > Labels: performance > Fix For: 0.9 > > Attachments: MAHOUT-1464.patch, MAHOUT-1464.patch, MAHOUT-1464.patch > > > Create a version of RowSimilarityJob that runs on Spark. Ssc has a prototype > here: https://gist.github.com/sscdotopen/8314254. This should be compatible > with Mahout Spark DRM DSL so a DRM can be used as input. > Ideally this would extend to cover MAHOUT-1422 which is a feature request for > RSJ on two inputs to calculate the similarity of rows of one DRM with those > of another. This cross-similarity has several applications including > cross-action recommendations. -- This message was sent by Atlassian JIRA (v6.2#6252)