On Wed, Jan 20, 2010 at 8:18 AM, Douglas Bates <[email protected]> wrote: > On Tue, Jan 19, 2010 at 9:26 AM, Martin Maechler > <[email protected]> wrote: >> Scanning for 'Matrix' in old R-help e-mail, I found >> >>>>>>> "GA" == Gad Abraham <[email protected]> >>>>>>> on Fri, 27 Nov 2009 13:45:00 +1100 writes: >> >> GA> Hi, >> GA> I'd like to store large covariance matrices using Matrix classes. >> >> GA> dsyMatrix seems like the right one, but I want to specify just the >> GA> upper/lower triangle and diagonal and not have to instantiate a huge >> GA> n^2 vector just for the sake of having half of it ignored: >> >> GA> Dumb example: >> GA> M <- new("dsyMatrix", uplo="U", x=rnorm(1e4), Dim=as.integer(c(100, >> 100))) >> GA> diag(M) <- 1 >> >> GA> This doesn't work: >> GA> M <- new("dsyMatrix", uplo="U", x=0, Dim=as.integer(c(100, 100))) >> GA> Error in validObject(.Object) : >> GA> invalid class "dsyMatrix" object: length of x slot != prod(Dim) >> >> GA> Is there an easier way of doing this? >> >> I think you want a "dspMatrix" ("sp" == "symmetric packed") >> instead. >> >> Before going into details: Is this topic still interesting to >> those involved almost two months ago? >> >> Regards, >> Martin > > Also, I fail to understand the advantage of allocating storage for > roughly half the elements in the matrix instead of the whole matrix > when the matrix is so large. If dense storage of a symmetric matrix > of size 20000 takes up too much memory (approx 3 GB for each copy) it > is unlikely that the packed symmetric storage form, using about 1.5 GB > per copy, is going to save the day. > >> 8 * (20000 * 20000)/2^30 # size in gigabytes for full array > [1] 2.980232 >> 4 * (20000 * 20001)/2^30 # size in gigabytes for sp array > [1] 1.490191
The difference between 16GB and 32 GB will determine whether I can store the matrix in RAM or not. > > If the matrix is sparse, the dsCMatrix class may help. It's a covariance matrix, it's dense. > > And there is also the issue of what exactly do you want to do with the > matrix once you have stored it? > I'm mainly interested in doing things like partial correlation using (taking the pseudoinverse of the matrix using SVD or Lanczos). -- Gad Abraham PhD Student, Dept. CSSE and NICTA The University of Melbourne Parkville 3010, Victoria, Australia email: [email protected] web: http://www.csse.unimelb.edu.au/~gabraham ______________________________________________ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
