On Thu, Aug 1, 2013 at 7:08 AM, Sebastian Schelter <s...@apache.org> wrote:
> IIRC the main reasons for deprecating Lanczos was that in contrast to > SSVD, it does not use a constant number of MapReduce jobs and that our > implementation has the constraint that all the resulting vectors have to > fit into the memory of the driver machine. > While it's true that Lanczos does not use a constant number of MR iterations, the phrase "our implementation" is key in saying we have to hold all the output vectors in memory. This wasn't even a very integral part of our impl. It's fairly simple to implement the linear combinations of the Ritz vectors after iterations are complete as an operation keeping only 3 vectors in memory at a time, we just never made that optimization. > > Best, > Sebastian > > On 01.08.2013 12:15, Fernando Fernández wrote: > > Hi everyone, > > > > Sorry if I duplicate the question but I've been looking for an answer > and I > > haven't found an explanation other than it's not being used (together > with > > some other algorithms). If it's been discussed in depth before maybe you > > can point me to some link with the discussion. > > > > I have successfully used Lanczos in several projects and it's been a > > surprise to me finding that the main reason (according to what I've read > > that might not be the full story) is that it's not being used. At the > > begining I supposed it was because SSVD is supposed to be much faster > with > > similar results, but after making some tests I have found that running > > times are similar or even worse than lanczos for some configurations (I > > have tried several combinations of parameters, given child processes > enough > > memory, etc. and had no success in running SSVD at least in 3/4 of time > > Lanczos runs, thouh they might be some combinations of parameters I have > > still not tried). It seems to be quite tricky to find a good combination > of > > parameters for SSVD and I have seen also a precision loss in some > examples > > that makes me not confident in migrating Lanczos to SSVD from now on (How > > far can I trust results from a combination of parameters that runs in > > significant less time, or at least a good time?). > > > > Can someone convince me that SSVD is actually a better option than > Lanczos? > > (I'm totally willing to be convinced... :) ) > > > > Thank you very much in advance. > > > > Fernando. > > > > -- -jake
