For 100,000 rows, it took about 2 seconds to read it in on my system: > system.time(x <- read.table('/recv/test.txt', as.is=TRUE)) user system elapsed 1.92 0.08 2.08 > str(x) 'data.frame': 196588 obs. of 7 variables: $ V1: int 1 2 3 4 1 2 3 1 2 3 ... $ V2: chr "bid" "ask" "ask" "bid" ... $ V3: chr "CON" "CON" "CON" "CON" ... $ V4: chr "09:30:00.722" "09:30:00.782" "09:30:00.809" "09:30:00.783" ... $ V5: chr "09:30:00.722" "09:30:00.810" "09:30:00.810" "09:30:00.810" ... $ V6: num 32.7 33.1 33.1 33.1 32.7 ... $ V7: int 98 300 414 200 98 300 414 98 300 414 ... > object.size(x) 6291928 bytes >
Given that you have about 85 files, I would guess that you would need about 800MB if all were 300K lines longs. You might be getting memory fragmentation. You might try using gc() every so often in the loop. What are you going to do with the data? Are you going to make one big file? In this case you might want a 64 bit version since you will have a single instance of 800K and will probably need 2-3X that much memory if copies are being made during processing. Object might be larger in 64-bit. Maybe you need to follow Gabor's advice and read it into a database and then process it from there. On Mon, Oct 11, 2010 at 5:48 PM, Gabor Grothendieck <ggrothendi...@gmail.com> wrote: > On Mon, Oct 11, 2010 at 5:39 PM, rivercode <aqua...@gmail.com> wrote: >> >> Hi, >> >> I am trying to find the best way to read 85 tick data files of format: >> >>> head(nbbo) >> 1 bid CON 09:30:00.722 09:30:00.722 32.71 98 >> 2 ask CON 09:30:00.782 09:30:00.810 33.14 300 >> 3 ask CON 09:30:00.809 09:30:00.810 33.14 414 >> 4 bid CON 09:30:00.783 09:30:00.810 33.06 200 >> >> Each file has between 100,000 to 300,300 rows. >> >> Currently doing nbbo.list<- lapply(filePath, read.csv) to create list >> with 85 data.frame objects...but it is taking minutes to read in the data >> and afterwards I get the following message on the console when taking >> further actions (though it does then stop): >> >> The R Engine is busy. Please wait, and try your command again later. >> >> filePath in the above example is a vector of filenames: >>> head(filePath) >> [1] "C:/work/A/A_2010-10-07_nbbo.csv" >> [2] "C:/work/AAPL/AAPL_2010-10-07_nbbo.csv" >> [3] "C:/work/ADBE/ADBE_2010-10-07_nbbo.csv" >> [4] "C:/work/ADI/ADI_2010-10-07_nbbo.csv" >> >> Is there a better/quicker or more R way of doing this ? >> > > You could try (possibly with suitable additonal arguments): > > library(sqldf) > lapply(filePath, read.csv.sql) > > -- > Statistics & Software Consulting > GKX Group, GKX Associates Inc. > tel: 1-877-GKX-GROUP > email: ggrothendieck at gmail.com > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve? ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.