[R] Reading large files quickly

I'm finding that readLines() and read.fwf() take nearly two hours to work through a 3.5 GB file, even when reading in large (100 MB) chunks. The unix command wc by contrast processes the same file in three minutes. Is there a faster way to read files in R? Thanks! __

Re: [R] Reading large files quickly

Thanks guys, good suggestions. To clarify, I'm running on a fast multi-core server with 16 GB RAM under 64 bit CentOS 5 and R 2.8.1. Paging shouldn't be an issue since I'm reading in chunks and not trying to store the whole file in memory at once. Thanks again. Rob Steele wrote

Re: [R] Reading large files quickly

meric? Are you keeping it in > a dataframe? Have you considered using 'scan' to read in the data and to > specify what the columns are? If you would like some more help, the answer > to these questions will help. > > On Sat, May 9, 2009 at 10:09 PM, Rob Steele >

Re: [R] Reading large files quickly; resolved

Rob Steele wrote: > I'm finding that readLines() and read.fwf() take nearly two hours to > work through a 3.5 GB file, even when reading in large (100 MB) chunks. > The unix command wc by contrast processes the same file in three > minutes. Is there a faster way to read files