On Tue, 9 Dec 2003 [EMAIL PROTECTED] wrote: > I would also like some clarification about R memory management. Like Doug, > I didn't find anything about consecutive calls to gc() to free more memory.
It was a statement about Windows, and about freeing memory *to Windows*. Douglas Grove apparently had misread both the subject line and the sentence. > We run into memory limit problems every now and then and a better > understanding of R's memory management would go a long way. I am interested > in learning more and was wondering if there is any specific R documentation > that explains R's memory usage? Or maybe some good links about memory and > garbage collection. Thanks. There are lots of comments in the source files. And as I already said (but has been excised below), this is not relevant to the next version of R anyway. BTW, the message below has been selectively edited, so please consult the original. > Message: 21 > Date: Mon, 8 Dec 2003 09:51:12 -0800 (PST) > From: Douglas Grove <[EMAIL PROTECTED]> > Subject: Re: [R] Windows Memory Issues > To: Prof Brian Ripley <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Message-ID: > <[EMAIL PROTECTED]> > Content-Type: TEXT/PLAIN; charset=US-ASCII > > On Sat, 6 Dec 2003, Prof Brian Ripley wrote: > > > I think you misunderstand how R uses memory. gc() does not free up all > > the memory used for the objects it frees, and repeated calls will free > > more. Don't speculate about how memory management works: do your > > homework! > > Are you saying that consecutive calls to gc() will free more memory than > a single call, or am I misunderstanding? Reading ?gc and ?Memory I don't > see anything about this mentioned. Where should I be looking to find > more comprehensive info on R's memory management?? I'm not writing any > packages, just would like to have a better handle on efficiently using > memory as it is usually the limiting factor with R. FYI, I'm running > R1.8.1 and RedHat9 on a P4 with 2GB of RAM in case there is any platform > specific info that may be applicable. > > Thanks, > > Doug Grove > Statistical Research Associate > Fred Hutchinson Cancer Research Center > > > > In any case, you are using an outdated version of R, and your first > > course of action should be to compile up R-devel and try that, as there > > has been improvements to memory management under Windows. You could also > > try compiling using the native malloc (and that *is* described in the > > INSTALL file) as that has different compromises. > > > > > > On Sat, 6 Dec 2003, Richard Pugh wrote: > > > > > Hi all, > > > > > > I am currently building an application based on R 1.7.1 (+ compiled > > > C/C++ code + MySql + VB). I am building this application to work on 2 > > > different platforms (Windows XP Professional (500mb memory) and Windows > > > NT 4.0 with service pack 6 (1gb memory)). This is a very memory > > > intensive application performing sophisticated operations on "large" > > > matrices (typically 5000x1500 matrices). > > > > > > I have run into some issues regarding the way R handles its memory, > > > especially on NT. In particular, R does not seem able to recollect some > > > of the memory used following the creation and manipulation of large data > > > objects. For example, I have a function which receives a (large) > > > numeric matrix, matches against more data (maybe imported from MySql) > > > and returns a large list structure for further analysis. A typical call > > > may look like this . > > > > > > > myInputData <- matrix(sample(1:100, 7500000, T), nrow=5000) > > > > myPortfolio <- createPortfolio(myInputData) > > > > > > It seems I can only repeat this code process 2/3 times before I have to > > > restart R (to get the memory back). I use the same object names > > > (myInputData and myPortfolio) each time, so I am not create more large > > > objects .. > > > > > > I think the problems I have are illustrated with the following example > > > from a small R session . > > > > > > > # Memory usage for Rui process = 19,800 > > > > testData <- matrix(rnorm(10000000), 1000) # Create big matrix > > > > # Memory usage for Rgui process = 254,550k > > > > rm(testData) > > > > # Memory usage for Rgui process = 254,550k > > > > gc() > > > used (Mb) gc trigger (Mb) > > > Ncells 369277 9.9 667722 17.9 > > > Vcells 87650 0.7 24286664 185.3 > > > > # Memory usage for Rgui process = 20,200k > > > > > > In the above code, R cannot recollect all memory used, so the memory > > > usage increases from 19.8k to 20.2. However, the following example is > > > more typical of the environments I use . > > > > > > > # Memory 128,100k > > > > myTestData <- matrix(rnorm(10000000), 1000) > > > > # Memory 357,272k > > > > rm(myTestData) > > > > # Memory 357,272k > > > > gc() > > > used (Mb) gc trigger (Mb) > > > Ncells 478197 12.8 818163 21.9 > > > Vcells 9309525 71.1 31670210 241.7 > > > > # Memory 279,152k > > > > > > Here, the memory usage increases from 128.1k to 279.1k > > > > > > Could anyone point out what I could do to rectify this (if anything), or > > > generally what strategy I could take to improve this? > > > > > > Many thanks, > > > Rich. > > > > > > Mango Solutions > > > Tel : (01628) 418134 > > > Mob : (07967) 808091 > > > > > > > > > [[alternative HTML version deleted]] > > > > > > ______________________________________________ > > > [EMAIL PROTECTED] mailing list > > > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > > > > > > > > > > -- > > Brian D. Ripley, [EMAIL PROTECTED] > > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > > University of Oxford, Tel: +44 1865 272861 (self) > > 1 South Parks Road, +44 1865 272866 (PA) > > Oxford OX1 3TG, UK Fax: +44 1865 272595 > > > > ______________________________________________ > > [EMAIL PROTECTED] mailing list > > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > > > > ______________________________________________ > [EMAIL PROTECTED] mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > > -- Brian D. Ripley, [EMAIL PROTECTED] Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595 ______________________________________________ [EMAIL PROTECTED] mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help