Hi there, As Edward noted, if you ask a numerical analyst about matrix inversion, they will respond "you never invert a matrix, but you solve the linear system associated with the matrix". Linear system solving is usually done with iterative methods or matrix decompositions (as noted above). The reason why people avoid matrix inversion is because of its inherited poor numerical stability.
Best, Anastasios On Tue, Sep 27, 2016 at 8:42 AM, Edward Fine <edward.f...@gmail.com> wrote: > I have not found matrix inversion algorithms in Spark and I would be > surprised to see them. Except for matrices with very special structure > (like those nearly the identity), inverting and n*n matrix is slower than > O(n^2), which does not scale. Whenever a matrix is inverted, usually a > decomposition or a low rank approximation is used, just as Sean pointed > out. See further https://en.wikipedia.org/wiki/Computational_ > complexity_of_mathematical_operations#Matrix_algebra > or if you really want to dig into it > Stoer and Bulirsch http://www.springer.com/us/book/9780387954523 > > On Mon, Sep 26, 2016 at 11:00 PM Sean Owen <so...@cloudera.com> wrote: > >> I don't recall any code in Spark that computes a matrix inverse. There is >> code that solves linear systems Ax = b with a decomposition. For example >> from looking at the code recently, I think the regression implementation >> actually solves AtAx = Atb using a Cholesky decomposition. But, A = n x k, >> where n is large but k is smallish (number of features), so AtA is k x k >> and can be solved in-memory with a library. >> >> On Tue, Sep 27, 2016 at 3:05 AM, Cooper <ahmad.raban...@gmail.com> wrote: >> > How is the problem of large-scale matrix inversion approached in Apache >> Spark >> > ? >> > >> > This linear algebra operation is obviously the very base of a lot of >> other >> > algorithms (regression, classification, etc). However, I have not been >> able >> > to find a Spark API on parallel implementation of matrix inversion. Can >> you >> > please clarify approaching this operation on the Spark internals ? >> > >> > Here <http://ieeexplore.ieee.org/abstract/document/7562171/> is a >> paper on >> > the parallelized matrix inversion in Spark, however I am trying to use >> an >> > existing code instead of implementing one from scratch, if available. >> > >> > >> > >> > -- >> > View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/Large-scale-matrix-inverse-in-Spark-tp27796.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > --------------------------------------------------------------------- >> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> > >> >> -- -- Anastasios Zouzias <a...@zurich.ibm.com>
