I'm finding that readLines() and read.fwf() take nearly two hours to
work through a 3.5 GB file, even when reading in large (100 MB) chunks.
The unix command wc by contrast processes the same file in three
minutes. Is there a faster way to read files in R?
Thanks!
__
Thanks guys, good suggestions. To clarify, I'm running on a fast
multi-core server with 16 GB RAM under 64 bit CentOS 5 and R 2.8.1.
Paging shouldn't be an issue since I'm reading in chunks and not trying
to store the whole file in memory at once. Thanks again.
Rob Steele wrote
meric? Are you keeping it in
> a dataframe? Have you considered using 'scan' to read in the data and to
> specify what the columns are? If you would like some more help, the answer
> to these questions will help.
>
> On Sat, May 9, 2009 at 10:09 PM, Rob Steele
>
Rob Steele wrote:
> I'm finding that readLines() and read.fwf() take nearly two hours to
> work through a 3.5 GB file, even when reading in large (100 MB) chunks.
> The unix command wc by contrast processes the same file in three
> minutes. Is there a faster way to read files
4 matches
Mail list logo