>>>>> Karl Millar <kmil...@google.com> >>>>> on Mon, 29 Feb 2016 10:22:51 -0800 writes:
> Thanks. > Couldn't you implement model.matrix(..., sparse = TRUE) with a small > amount of R code similar to MatrixModels::model.Matrix ? yes, and basically call R level Matrix::sparse.model.matrix() [[ or even just mention the latter on the help page for model.matrix() ]]. Thank you, Karl > On Mon, Feb 29, 2016 at 10:01 AM, Martin Maechler > <maech...@stat.math.ethz.ch> wrote: >>>>>>> Karl Millar via R-devel <r-devel@r-project.org> >>>>>>> on Fri, 26 Feb 2016 15:58:20 -0800 writes: >> >> > Generating a model matrix with very large numbers of >> > columns overflows the stack and/or runs very slowly, due >> > to the implementation of TrimRepeats(). >> >> > This patch modifies it to use Rf_duplicated() to find the >> > duplicates. This makes the running time linear in the >> > number of columns and eliminates the recursive function >> > calls. >> >> Thank you, Karl. >> I've committed this (very slightly modified) to R-devel, >> >> (also after looking for a an example that runs on a non-huge >> computer and shows the difference) : >> >> nF <- 11 ; set.seed(1) >> lff <- setNames(replicate(nF, as.factor(rpois(128, 1/4)), simplify=FALSE), letters[1:nF]) >> str(dd <- as.data.frame(lff)); prod(sapply(dd, nlevels)) >> ## 'data.frame': 128 obs. of 11 variables: >> ## $ a: Factor w/ 3 levels "0","1","2": 1 1 1 2 1 2 2 1 1 1 ... >> ## $ b: Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 2 1 1 1 ... >> ## $ c: Factor w/ 3 levels "0","1","2": 1 1 1 2 1 1 1 2 1 1 ... >> ## $ d: Factor w/ 3 levels "0","1","2": 1 1 2 2 1 2 1 1 2 1 ... >> ## $ e: Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 2 1 ... >> ## $ f: Factor w/ 2 levels "0","1": 2 1 2 1 2 1 1 2 1 2 ... >> ## $ g: Factor w/ 4 levels "0","1","2","3": 2 1 1 2 1 3 1 1 1 1 ... >> ## $ h: Factor w/ 4 levels "0","1","2","4": 1 1 1 1 2 1 1 1 1 1 ... >> ## $ i: Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 2 ... >> ## $ j: Factor w/ 3 levels "0","1","2": 1 2 3 1 1 1 1 1 1 1 ... >> ## $ k: Factor w/ 3 levels "0","1","2": 1 1 1 1 1 1 1 1 1 1 ... >> ## >> ## [1] 139968 >> >> system.time(mff <- model.matrix(~ . ^ 11, dd, contrasts = list(a = "contr.helmert"))) >> ## user system elapsed >> ## 0.255 0.033 0.287 --- *with* the patch on my desktop (16 GB) >> ## 1.489 0.031 1.522 --- for R-patched (i.e. w/o the patch) >> >>> dim(mff) >> [1] 128 139968 >>> object.size(mff) >> 154791504 bytes >> >> --- >> >> BTW: These example would gain tremendously if I finally got >> around to provide >> >> model.matrix(........, sparse = TRUE) >> >> which would then produce a Matrix-package sparse matrix. >> >> Even for this somewhat small case, a sparse matrix is a factor >> of 13.5 x smaller : >> >>> s1 <- object.size(mff); s2 <- object.size(M <- Matrix::Matrix(mff)); as.vector( s1/s2 ) >> [1] 13.47043 >> >> I'm happy to collaborate with you on adding such a (C level) >> interface to sparse matrices for this case. >> >> Martin Maechler ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel