[
https://issues.apache.org/jira/browse/MAHOUT-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159469#comment-13159469
]
Dmitriy Lyubimov edited comment on MAHOUT-797 at 11/29/11 7:38 PM:
-------------------------------------------------------------------
so what i think is that using R'^-1YA route is more promising overall for B_0
pipeline.
The reason being that the version that uses explicit Q must pull in both Q and
A while the version that would use R^-1 would only go over A and Omega, but
Omega is imaginary. So we trade slightly more flops for less network I/O. This
route may turn out significantly more beneficial if we believe that size of Q
>> size of A. I imagine this is not too common though, but i have high hopes.
we may eventually even deploy some automatic default based on subsampling A for
sparsity.
For power iterations pipeline perhaps looses some of it since we don't have
random matrix in the works anymore. It seems possible that i might be able to
do both Y_i = AB'_i-1 and Y'_i Y_i in one MR huge step and then pull in Y_i
as a side info, but it would require producing Y output the same way as A is
split which is technically possible (similarly how it is done with map-side
first qr step).
But all of that changes the flow significantly enough to practically build a
completely new set of jobs (which is ok).
was (Author: dlyubimov):
so what i think is that using R'^-1YA route is more promising overall for
B_0 pipeline.
The reason being that the version that uses explicit Q must pull in both Q and
A while the version that would use R^-1 would only go over A and Omega, but
Omega is imaginary. So we trade slightly more flops for less network I/O. This
route may turn out significantly more beneficial if we believe that size of Q
>> size of A. I imagine this is not too common though, but i have high hopes.
we may eventually even deploy some automatic default based on subsampling A for
sparsity.
For power iterations pipeline perhaps looses some of it since we don't have
random matrix in the works anymore. It seems possible that i might be able to
do both Y_i = AB'_i-1 and Y'_i Y_i in one MR step there without ever forming
AB' and that would be a huge plus instead.
But all of that changes the flow significantly enough to practically build a
completely new set of jobs (which is ok).
> MapReduce SSVD: provide alternative B-pipeline per B=R' ^{-1} Y'A
> -----------------------------------------------------------------
>
> Key: MAHOUT-797
> URL: https://issues.apache.org/jira/browse/MAHOUT-797
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5, 0.6
> Reporter: Dmitriy Lyubimov
> Assignee: Dmitriy Lyubimov
> Fix For: Backlog
>
> Attachments: MAHOUT-797.pdf
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Since alternative flow using Cholesky decomposition is extremely easy to add
> to existing computations of BB', I am thinking of just adding an option that
> chooses between B-pipeline with QR step and B-pipeline with Y'Y-Cholesky
> step.
> Ongoing work and some initial code (Y'Y step) is here
> https://github.com/dlyubimov/mahout-commits/tree/MAHOUT-797.
> I also want to fix what's left unfixed in MAHOUT-638.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
