Hi Gabor: Thanks. My files are all in fixed width format. They are a lot of them. It would take me some effort to convert them to CSV. I guess this cannot be avoided? I can write some Perl scripts to convert fixed width format to CSV format and then start with your suggestion. Could you let me know your thoughts on the approach? Satish
-----Original Message----- From: Gabor Grothendieck [mailto:ggrothendi...@gmail.com] Sent: Friday, February 05, 2010 5:16 PM To: Vadlamani, Satish {FLNA} Cc: r-help@r-project.org Subject: Re: [R] Reading large files If your problem is just how long it takes to load the file into R try read.csv.sql in the sqldf package. A single read.csv.sql call can create an SQLite database and table layout for you, read the file into the database (without going through R so R can't slow this down), extract all or a portion into R based on the sql argument you give it and then remove the database. See the examples on the home page: http://code.google.com/p/sqldf/#Example_13._read.csv.sql_and_read.csv2.sql On Fri, Feb 5, 2010 at 2:11 PM, Satish Vadlamani <satish.vadlam...@fritolay.com> wrote: > > Matthew: > If it is going to help, here is the explanation. I have an end state in > mind. It is given below under "End State" header. In order to get there, I > need to start somewhere right? I started with a 850 MB file and could not > load in what I think is reasonable time (I waited for an hour). > > There are references to 64 bit. How will that help? It is a 4GB RAM machine > and there is no paging activity when loading the 850 MB file. > > I have seen other threads on the same types of questions. I did not see any > clear cut answers or errors that I could have been making in the process. If > I am missing something, please let me know. Thanks. > Satish > > > End State >> Satish wrote: "at one time I will need to load say 15GB into R" > > > ----- > Satish Vadlamani > -- > View this message in context: > http://n4.nabble.com/Reading-large-files-tp1469691p1470667.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.