Re: [R] Need advice on using R with large datasets

[Paul Gilbert]

Liaw, Andy wrote:

I was under the impression that R has been run on 64-bit Solaris (and other
64-bit Unices) for quite a while (as 64-bit app).  

Yes, on Solaris it has worked for quite a while. I don't use it a lot, 
but have one problem that I have been running from time to time for a 
few years.  There are two "issues" that I know about.

1/  Some extra capabilities (like png I think)  also need to be compiled 
as 64 bit apps, and in some cases this is a non-trivial effort (on 
Solaris for someone like me that does not do that kind of thing often). 
For this reason I have both a 32-bit version for regular use and a 
64-bit version for special problems.

2/  Some R functions make copies of the data sets used and attach them 
to the result. For small data sets that can be very useful. If the 
result is then used as an argument to another function then very quickly 
there are multiple copies. If the data set is large then one is quickly 
making heavy use of swap, and the processing is very slow. This is not 
just a 64-bit problem, but with a 32-bit architecture it is hard to work 
on a data set big enough that this becomes an issue.  In some cases 
performance  can be improved  a lot by hacking the code and  not 
attaching the  dataset to the result (with some risk that functions 
using the result get broken).

Paul Gilbert

We've been running 64-bit
R on amd64 for a few months (and had quite a few oppertunities to get the R
processes using over 8GB of RAM).  Not much problem as far as I can see...
Best,
Andy
 

From: Roger D. Peng

As far as I know, R does compile on AMD Opterons and runs as a 
64-bit application.  So it can store objects larger than 4GB. 
However, I don't think R gets tested very often on 64-bit 
machines with such large objects so there may be yet undiscovered 
bugs.

-roger

Sunny Ho wrote:

   

Hello everyone,

I would like to get some advices on using R with some 
     

really large datasets.
   

I'm using RH9 Linux R 1.8.1 for a research with a lot of 
     

numerical data. The datasets total to around 200Mb (shown by 
memory.size). During my data manipulation, the system memory 
usage grew to 1.5Gb, and this caused a lot of swapping 
activities on my 1Gb PC. This is just a small-scale 
experiment, the full-scale one will be using data 30 times as 
large (on a 4Gb machine). I can see that I'll need to deal 
with memory usage problem very soon.
   

I notice that R keeps all datasets in memory at all times. 
     

I wonder whether there is any way to instruct R to push some 
of the less-frequently-used data tables out of main memory, 
so as to free up memory for those that are actively in used. 
It'll be even better if R can keep only part of a table in 
memory only when that part is needed. Using save & load could 
help, but I just wonder whether R is intelligent enough to do 
this by itself, so I don't need to keep track of memory usage 
at all times.
   

Another thought is to use a 64-bit machine (AMD64). I find 
     

there is a pre-compiled R for Fedora Linux on AMD64. Anyone 
knows whether this version of R runs as 64-bit? If so, then 
will R be able to go beyond the 32-bit 4Gb memory limit?
   

Also, from the manual, I find that the RPgSQL package (for 
     

PostgreSQL database) supports a feature "proxy data frame". 
Does anyone have experience with this? Can "proxy data frame" 
handle memory efficiently for very large datasets? Say, if I 
have a 6Gb database table defined as a proxy data frame, will 
R & RPgSQL be able to handle it with just 4Gb of memory?
   

Any comments will be useful. Many thanks.

Sunny Ho
(Hong Kong University of Science & Technology)
______________________________________________
[EMAIL PROTECTED] mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! 
     

http://www.R-project.org/posting-guide.html
   

______________________________________________
[EMAIL PROTECTED] mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! 
http://www.R-project.org/posting-guide.html

   



------------------------------------------------------------------------------
Notice:  This e-mail message, together with any attachments,...{{dropped}}
______________________________________________
[EMAIL PROTECTED] mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
 

______________________________________________
[EMAIL PROTECTED] mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html