Just an update. I am still working out a version that would do 100% mapreduce version of Q orhonormalization . I think i made some progress with the prototype, and basically there seems also to be a way to drastically cut on the output of the first map reduce (i.e. reduce amount of partial B^t outer products) depending on how much one wants to compromise the stochastic effect produced by Y. The algorithm also significantly simplifies the orthogonolization process by basically getting rid of Y-blocking entirely inside the same mapper (i.e. it's 100% streaming, 1 Q-block per mapper with memory requirements still constant to size of a row of A).
It's slow due to my family situtation (i had just several hours to work on non-mr prototype so far, which passed Q orthogonality test, everything else is just a thought experiment) On Sun, Oct 24, 2010 at 12:11 AM, Ted Dunning (JIRA) <[email protected]>wrote: > > [ > https://issues.apache.org/jira/browse/MAHOUT-376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] > > Ted Dunning updated MAHOUT-376: > ------------------------------- > > Attachment: sd.tex > sd.pdf > > Updated version. > > I think that this is actually a feasible algorithm. > > > Implement Map-reduce version of stochastic SVD > > ---------------------------------------------- > > > > Key: MAHOUT-376 > > URL: https://issues.apache.org/jira/browse/MAHOUT-376 > > Project: Mahout > > Issue Type: Improvement > > Components: Math > > Reporter: Ted Dunning > > Assignee: Ted Dunning > > Fix For: 0.5 > > > > Attachments: MAHOUT-376.patch, Modified stochastic svd algorithm > for mapreduce.pdf, sd-bib.bib, sd.pdf, sd.pdf, sd.pdf, sd.pdf, sd.tex, > sd.tex, sd.tex, sd.tex, Stochastic SVD using eigensolver trick.pdf > > > > > > See attached pdf for outline of proposed method. > > All comments are welcome. > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > >
