Hello R List,
I'm trying to run a linear mixed effects analysis on fairly large datasets with
lmer (from the lme4 package) on a 32-bit Windows machine running XP with 3 GB
of RAM. It's not working. (details below)
I've researched the ff and bigmemory packages, but it appears they won't handle
the mixed mode dataset I'm analyzing. It has some character fields for the
categorical variables. It's also not clear a linear mixed effect regression can
be run with those packages. Can anyone point me to a lme solution on larger
datasets that can address the 32-bit memory limitations?
The smallest dataframe I'm trying to run is about 120,000 observations and 7
variables, but I'd rather run a for loop script on 620K+ observations. I've
seen several recommendations (Kabacoff in 'R in Action' and others) that when
possible, run R in a 64-bit build. Problem is I'm on a deadline, and procuring
a new computer takes time, and approvals up the food chain. Suggestions? Is
64-bit my only option?
Here is the information about what I'm running:
R version 2.14.0 (2011-10-31)
Platform: i386-pc-mingw32/i386 (32-bit)
locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods
[7] base
other attached packages:
[1] lme4_0.999999-0 Matrix_1.0-5 lattice_0.20-10
[4] RODBC_1.3-6
loaded via a namespace (and not attached):
[1] grid_2.14.0 nlme_3.1-104 stats4_2.14.0 tools_2.14.0
I run this code:
> install.packages("lme4")
> library(lme4)
> math07g4 <- sqlQuery(conn, "select ssid, ss_chg, campus2, district_id,
> pblack, pfreelnch, pmob
FROM codemob0607ma WHERE grade2 = 4")
> attach(math07g4)
> fit07ma4 <- lmer(ss_chg ~ 1 + factor(campus2) + factor(district_id) + pblack
> + pfreelnch + pmob +
(1 | campus2) + (1 | district_id), data=math07g4)
And I get this:
Error: cannot allocate vector of size 2.5 Gb
In addition: Warning messages:
1: In model.matrix.default(mt, mf, contrasts) :
Reached total allocation of 2187Mb: see help(memory.size)
2: In model.matrix.default(mt, mf, contrasts) :
Reached total allocation of 2187Mb: see help(memory.size)
3: In model.matrix.default(mt, mf, contrasts) :
Reached total allocation of 2187Mb: see help(memory.size)
4: In model.matrix.default(mt, mf, contrasts) :
Reached total allocation of 2187Mb: see help(memory.size)
Thanks in advance,
Ken
Ken McCall | Database reporter
Dayton Daily News<http://www.daytondailynews.com/> | 1611 S. Main St. Dayton,
Ohio 45409
Ph: 937-000-000 Cell: 937-225-2393 Email:
ken.mcc...@coxinc.com<mailto:ken.mcc...@coxinc.com>
COMPLETE. IN-DEPTH. DEPENDABLE.
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
