Re: [R] Once again: Error: cannot allocate vector of size
Please re-read the posting guide e.g. you didn't provide an example data set or a way to generate one, or any R version information. Werner W. pensterfuz...@yahoo.de wrote in message news:646146.32238...@web23002.mail.ird.yahoo.com... Hi, I have browsed the help list and looked at the FAQ but I don't find conclusive evidence if this is normal or I am doing something wrong. I am running a lm() on a data.frame with 27136 observations of 6 variables (3 num and 3 factor). After a while R throws this: lm(log(y) ~ log(a) + log(b) + c + d + e, data=reg.data , na.action=na.exclude) Error: cannot allocate vector of size 203.7 MB This is a Windows XP 32 bit machine with 4 GB in it so that theoretically, R should be able to claim close to 2 GB. This is the gc() after the regression: used (Mb) gc trigger (Mb) max used (Mb) Ncells 272299 7.3 875833 23.4 1368491 36.6 Vcells 4526037 34.6 116536251 889.2 145524997 1110.3 memory.size(max=T) [1] 1230.25 memory.size(max=F) [1] 47.89 Looking at memory.size, R should be easily able to allocate that space, shouldn't it? Many thanks for any hints! --Werner __ Do You Yahoo!? Sie si n Massenmails. http://mail.yahoo.com __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Once again: Error: cannot allocate vector of size
Hi Werner, On Thu, Jan 21, 2010 at 9:23 PM, Werner W. pensterfuz...@yahoo.de wrote: Hi, I have browsed the help list and looked at the FAQ but I don't find conclusive evidence if this is normal or I am doing something wrong. I am running a lm() on a data.frame with 27136 observations of 6 variables (3 num and 3 factor). After a while R throws this: lm(log(y) ~ log(a) + log(b) + c + d + e, data=reg.data , na.action=na.exclude) Error: cannot allocate vector of size 203.7 MB This is a Windows XP 32 bit machine with 4 GB in it so that theoretically, R should be able to claim close to 2 GB. This is the gc() after the regression: used (Mb) gc trigger (Mb) max used (Mb) Ncells 272299 7.3 875833 23.4 1368491 36.6 Vcells 4526037 34.6 116536251 889.2 145524997 1110.3 memory.size(max=T) [1] 1230.25 memory.size(max=F) [1] 47.89 Looking at memory.size, R should be easily able to allocate that space, shouldn't it? AFAIK, the error message Error: cannot allocate vector of size 203.7 MB doesn't mean that your entire data set is 203.7MB and it can't load it ... but rather R just tried grab another chunk of RAM that size, and the OS has denied it that favor ... so, I'm guessing it's getting half way through loading/running on your data, but can't complete the task because it has already chewed through all the RAM that at has access to. You can look at your RAM usage (through the task manager? sorry, haven't used windows in a long time) while R is doing its thing to verify until a windows guru chimes in. -steve -- Steve Lianoglou Graduate Student: Computational Systems Biology | Memorial Sloan-Kettering Cancer Center | Weill Medical College of Cornell University Contact Info: http://cbio.mskcc.org/~lianos/contact __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Once again: Error: cannot allocate vector of size
Thanks Matthew, you are absolutely right. I am working on Windows XP SP2 32bit with R versions 2.9.1. Here is an example: d - as.data.frame(matrix(trunc(rnorm(6*27136, 1, 100)),ncol=6)) d[,4:5] - trunc(100*runif(2*27136, 0, 1)) d[,6] - trunc(1000*runif(27136, 0, 1)) for (i in 4:6) d[,i] - as.factor(d[,i]) lm(V1 ~ log(V2) + log(V3) + V4 + V5 + V6, data=d) memory.size(max=F) memory.size(max=T) I managed to get it run through after setting the 3GB switch for Windows and with a clean R session. I also noticed later, that after removing na.action=na.exclude more regressions run through. But before and after the lm() it seems there should be enough memory which means that lm() builds up some quite large objects during its computations? -- View this message in context: http://n4.nabble.com/Once-again-Error-cannot-allocate-vector-of-size-tp1083506p1100164.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Once again: Error: cannot allocate vector of size
Fantastic. You're much more likely to get a response now. Best of luck. werner w pensterfuz...@yahoo.de wrote in message news:1264175935970-1100164.p...@n4.nabble.com... Thanks Matthew, you are absolutely right. I am working on Windows XP SP2 32bit with R versions 2.9.1. Here is an example: d - as.data.frame(matrix(trunc(rnorm(6*27136, 1, 100)),ncol=6)) d[,4:5] - trunc(100*runif(2*27136, 0, 1)) d[,6] - trunc(1000*runif(27136, 0, 1)) for (i in 4:6) d[,i] - as.factor(d[,i]) lm(V1 ~ log(V2) + log(V3) + V4 + V5 + V6, data=d) memory.size(max=F) memory.size(max=T) I managed to get it run through after setting the 3GB switch for Windows and with a clean R session. I also noticed later, that after removing na.action=na.exclude more regressions run through. But before and after the lm() it seems there should be enough memory which means that lm() builds up some quite large objects during its computations? -- View this message in context: http://n4.nabble.com/Once-again-Error-cannot-allocate-vector-of-size-tp1083506p1100164.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Once again: Error: cannot allocate vector of size
The problem I see here is not so much the 27136 observations, but the fact that the first two factors have up to 101 different levels and the third factor up to 1001 different levels. That means that lm() is essentially creating 100+100+1000 dummies for those factors, leading to a large (and sparse) design matrix (containing over 30 million entries). Best, -- Wolfgang Viechtbauerhttp://www.wvbauer.com/ Department of Methodology and StatisticsTel: +31 (43) 388-2277 School for Public Health and Primary Care Office Location: Maastricht University, P.O. Box 616 Room B2.01 (second floor) 6200 MD Maastricht, The Netherlands Debyeplein 1 (Randwyck) Original Message From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Matthew Dowle Sent: Friday, January 22, 2010 17:33 To: r-h...@stat.math.ethz.ch Subject: Re: [R] Once again: Error: cannot allocate vector of size Fantastic. You're much more likely to get a response now. Best of luck. werner w pensterfuz...@yahoo.de wrote in message news:1264175935970-1100164.p...@n4.nabble.com... Thanks Matthew, you are absolutely right. I am working on Windows XP SP2 32bit with R versions 2.9.1. Here is an example: d - as.data.frame(matrix(trunc(rnorm(6*27136, 1, 100)),ncol=6)) d[,4:5] - trunc(100*runif(2*27136, 0, 1)) d[,6] - trunc(1000*runif(27136, 0, 1)) for (i in 4:6) d[,i] - as.factor(d[,i]) lm(V1 ~ log(V2) + log(V3) + V4 + V5 + V6, data=d) memory.size(max=F) memory.size(max=T) I managed to get it run through after setting the 3GB switch for Windows and with a clean R session. I also noticed later, that after removing na.action=na.exclude more regressions run through. But before and after the lm() it seems there should be enough memory which means that lm() builds up some quite large objects during its computations? -- View this message in context: http://n4.nabble.com/Once-again-Error-cannot-allocate-vector-of-size-tp1083506p1100164.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
