Thank you Uwe, John, and Bert - this is very helpful context. If it helps inform the discussion, to address John and Bert's questions - I actually had less memory free when I originally ran the analyses and saved the workspace, than when I read in the data back in later on (I rebooted in an attempt to free all possible memory before rereading the workspace back in).
On Wed, Sep 2, 2020 at 1:27 PM John via R-help <r-help using r-project.org> wrote: >> On Wed, 2 Sep 2020 13:36:43 +0200 >> Uwe Ligges <ligges using statistik.tu-dortmund.de> wrote: >> >> > On 02.09.2020 04:44, David Jones wrote: >> > > I ran a number of analyses in R and saved the workspace, which >> > > resulted in a 2GB .RData file. When I try to read the file back >> > > into R >> > >> > Compressed in RData but uncompressed in main memory.... >> > >> > >> > > later, it won't read into R and provides the error: "Error: cannot >> > > allocate vector of size 37 Kb" >> > > >> > > This error comes after 1 minute of trying to read things in - I >> > > presume a single vector sends it over the memory limit. But, >> > > memory.limit() shows that I have access to a full 16gb of ram on my >> > > machine (12 GB are free when I try to load the RData file). >> > >> > But the data may need more.... >> > >> > >> > > gc() shows the following after I receive this error: >> > > >> > > used (Mb) gc trigger (Mb) max used (Mb) >> > > Ncells 623130 33.3 4134347 220.8 5715387 305.3 >> > > Vcells 1535682 11.8 883084810 6737.5 2100594002 16026.3 >> > >> > So 16GB were used when R gave up. >> > >> > Best, >> > Uwe Ligges >> >> For my own part, looking at the OP's question, it does seem curious >> that R could write that .RData file, but on the same system not be able >> to reload something it created. How would that work. Wouldn't the >> memory limit have been exceeded BEFORE the the .RData file was written >> the FIRST time? >> >> JDougherty >R experts may give you a detailed explanation, but it is certainly possible >that the memory available to R when it wrote the file was different than >when it tried to read it, is it not? >Bert Gunter >"The trouble with having an open mind is that people keep coming along and >sticking things into it." >-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) ______________________________________________ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
