Thanks all for your helpful suggestions. I replaced the hard drive of this computer and augmented the RAM to 8 GB over the weekend, and things seen to be running very smoothly now. I'll certainly use the methods provided here as I push the limits on these types of matrices and objects.
Best *Ben Caldwell* On Sun, Apr 21, 2013 at 9:21 AM, Charles Berry <ccbe...@ucsd.edu> wrote: > Benjamin Caldwell <btcaldwell <at> berkeley.edu> writes: > > > > > Dear R helpers > > > > Reproducible example: > > > > #warning - this causes a hard freeze on the machines I've tried it on > > matrix.holder<- matrix(rnorm(150), nrow=30, ncol=5) > > > > Out= > > expand.grid(matrix.holder[,1],matrix.holder[,2],matrix.holder[,3], > matrix.holder[,4], > > matrix.holder[,5]) > > > > Problem: > > > > I'm running an analysis that I would like to do using a matrix containing > > all the possible combinations of the elements in a [30,5] matrix. > Briefly, > > each possible combination is used to index and subset another matrix. I > > then run some models on the data in the subsetted matrix and then > > sometimes > > export the model results based on a couple criteria. 24,300,000 > > combinations seems to be too big for R on my computer (Intel i5, about > 2.5 > > GB RAM free, 4 GB total, Rx64 2.15 ) to handle. > > > > Requests: > > > [snip] > > > > > I'd like to attempt to multithread [snip] > > > Ben, > > The problem you have is "embarassingly parallel" - as they say. > > You can effectively use brute force solutions to parallelize the job > and do it with subjobs that have smaller memory requirements. > > One way to parallelize the problem is to create the object 'matrix.holder', > then loop thru the values of matrix.holder[,1] and create a subjob that > will run all the computations for matrix.holder[i,1] and all the > combinations of matrix.holder[,-1]. Run the subjob in a new process and > save > the results. Later on you combine the saved results. > > Also, you could try to run each subjob using parallel::mclapply() or > some other parallelizing package. Or you could loop over each of the > first two columns of matrix.holder creating 900 subjobs. Also, this gives > you still smaller memory requirements for the individual jobs. > > HTH, > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
