[R] how to append new data to saved data on disk efficiently

2009-12-29 Thread Hao Cen
Hi,

I currently combine multiple processed data (data frame) into a list and
save the list as .rda using the save command. When new data come, I load
the rda file, process the new data into a data frame, append the data
frame to the end of the list, and save the whole list to the disk. The
loading and saving steps are quite time consuming. Since I don't need to
change the old data in the list when new data come, I wonder if there is
an efficient way to update the rda efficiently, e.g. appending the new
data to the rda file directly without loading the whole rda file?

Any suggestions would be appreciated.

thanks

Jeff

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


Re: [R] how to append new data to saved data on disk efficiently

2009-12-29 Thread jim holtman
You can keep each dataframe as a separate file and process them that way.
You can look into storing in a relational database or using filehash.  It
all depends on how you want to process the data later.

On Tue, Dec 29, 2009 at 8:40 AM, Hao Cen h...@andrew.cmu.edu wrote:

 Hi,

 I currently combine multiple processed data (data frame) into a list and
 save the list as .rda using the save command. When new data come, I load
 the rda file, process the new data into a data frame, append the data
 frame to the end of the list, and save the whole list to the disk. The
 loading and saving steps are quite time consuming. Since I don't need to
 change the old data in the list when new data come, I wonder if there is
 an efficient way to update the rda efficiently, e.g. appending the new
 data to the rda file directly without loading the whole rda file?

 Any suggestions would be appreciated.

 thanks

 Jeff

 __
 R-help@r-project.org mailing list
 https://stat.ethz.ch/mailman/listinfo/r-help
 PLEASE do read the posting guide
 http://www.R-project.org/posting-guide.htmlhttp://www.r-project.org/posting-guide.html
 and provide commented, minimal, self-contained, reproducible code.




-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.