I think R will never do it's own memory swapping, as that is a typical OS task. There are however several developments (provided in add-on packages) that will not load all data in memory at start-up, but instead call some data base whenever a data element is needed. You might search r-help for rsqlite or biglm, and there are others; also look at the award winners at useR this year.
Here, we've run pretty successful R sessions needing 10-11 Gb of memory on a 8Gb RAM 64 bits linux machine with lots of swap space. Needs some patience, and still R might crash other parts of the system when memory usage becomes too excessive. Best regards, -- Edzer Didier Leibovici wrote: > Thanks Roger > > I feel we've got a low RAM machine which would need a bit of an uplift > (recent server though)! > The linux machine is unfortunately also with 4Gb of RAM > But I persist to say it would be interesting to have within R a way of > automatically performing swapping memory if needed ... > > Didier > > Roger Bivand wrote: > >> On Tue, 11 Sep 2007, [EMAIL PROTECTED] wrote: >> >> >>>> These days in GIS on may have to manipulate big datasets or arrays. >>>> >>>> Here I am on WINDOWS I have a 4Gb >>>> my aim was to have an array of dim 298249 12 10 22 but that's 2.9Gb >>>> >> Assuming double precision (no single precision in R), 5.8Gb. >> >> >>> It used to be (maybe still is?) the case that a single process could >>> only >>> 'claim' a chunk of max size 2GB on Windows. >>> >>> >>> Also remember to compute overhead for R objects... 58 bytes per >>> object, I >>> think it is. >>> >>> >>> >>>> It is also strange that once a dd needed 300.4Mb and then 600.7Mb >>>> (?) as >>>> also I made some room in removing ZZ? >>>> >>> Approximately double size - many things the interpreter does involve >>> making an additional copy of the data and then working with *that*. >>> This >>> might be happening here, though I didn't read your code carefully enough >>> to be able to be certain. >>> >>> >>> >>>> which I don't really know if it took into account as the limit is >>>> greater than the physical RAM of 4GB. ...? >>>> >>> :) >>> >>> >>>> would it be easier using Linux ? >>>> >>> possibly a little bit - on a linux machine you can at least run a PAE >>> kernel (giving you a lot more address space to work with) and have the >>> ability to turn on a bit more virtual memory. >>> >>> usually with data of the size you're trying to work with, i try to >>> find a >>> way to preprocess the data a bit more before i apply R's tools to it. >>> sometimes we stick it into a database (postgres) and select out the bits >>> we want our inferences to be sourced from. ;) >>> >>> it might be simplest to just hunt up a machine with 8 or 16GB of >>> memory in >>> it, and run those bits of the analysis that really need memory on that >>> machine... >>> >> Yes, if there is no other way, a 64bit machine with lots of RAM would >> not be so contrained, but maybe this is a matter of first deciding why >> doing statistics on that much data is worth the effort? It may be, but >> just trying to read large amounts of data into memory is perhaps not >> justified in itself. >> >> Can you tile or subset the data, accumulating intermediate results? >> This is the approach the biglm package takes, and the R/GDAL interface >> also supports subsetting from an external file. >> >> Depending on the input format of the data, you should be able to do >> all you need provided that you do not try to keep all the data in >> memory. Using a database may be a good idea, or if the data are >> multiple remote sensing images, subsetting and accumulating results. >> >> Roger >> >> >>> --e >>> >>> _______________________________________________ >>> R-sig-Geo mailing list >>> R-sig-Geo@stat.math.ethz.ch >>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo >>> >>> > > > _______________________________________________ R-sig-Geo mailing list R-sig-Geo@stat.math.ethz.ch https://stat.ethz.ch/mailman/listinfo/r-sig-geo
