On Fri, 5 May 2006, Robert Citek wrote:
>
> On May 5, 2006, at 11:30 AM, Thomas Lumley wrote:
>> In addition to Uwe's message it is worth pointing out that gc()
>> reports
>> the maximum memory that your program has used (the rightmost two
>> columns).
>> You will probably see that this is large.
>
> Reloading the 10 MM dataset:
Ah, but the comment was about the 100 MM dataset, the one which gave you a
problem.
> R > foo <- read.delim("dataset.010MM.txt")
>
> R > object.size(foo)
> [1] 440000376
>
> R > gc()
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 10183941 272.0 15023450 401.2 10194267 272.3
> Vcells 20073146 153.2 53554505 408.6 50086180 382.2
>
> Combined, Ncells or Vcells appear to take up about 700 MB of RAM,
> which is about 25% of the 3 GB available under Linux on 32-bit
> architecture.
Please re-read help("Memory-limits"). You have a 3Gb address space, and
are looking for at least one 500Mb chunk. Fragmentation will come into
play here and it is quite likely that malloc will be unable to find a
500Mb chunk once you have allocated 1Gb. In attempting to read the
100 MM dataset you probably did go over 1Gb.
> Also, removing foo seemed to free up "used" memory, but didn't change
> the "max used":
Well, it doesn't change history does it? You were not expecting removing
objects to increase the memory used, I hope. From ?gc:
The final two columns show the maximum space used since the last
call to 'gc(reset=TRUE)' (or since R started).
> R > rm(foo)
>
> R > gc()
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 186694 5.0 12018759 321.0 10194457 272.3
> Vcells 74095 0.6 44173915 337.1 50085563 382.2
--
Brian D. Ripley, [EMAIL PROTECTED]
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
