Thanks for looking, but my file has quotes. It's also 400MB, and I don't mind waiting, but don't have 6x the memory to read it in.
On 8/9/07, Gabor Grothendieck <[EMAIL PROTECTED]> wrote: > > If we add quote = FALSE to the write.csv statement its twice as fast > reading it in. > > On 8/9/07, Michael Cassin <[EMAIL PROTECTED]> wrote: > > Hi, > > > > I've been having similar experiences and haven't been able to > > substantially improve the efficiency using the guidance in the I/O > > Manual. > > > > Could anyone advise on how to improve the following scan()? It is not > > based on my real file, please assume that I do need to read in > > characters, and can't do any pre-processing of the file, etc. > > > > ## Create Sample File > > write.csv(matrix(as.character(1:1e6),ncol=10,byrow=TRUE),"big.csv", > row.names=FALSE) > > q() > > > > **New Session** > > #R > > system("ls -l big.csv") > > system("free -m") > > big1<-matrix(scan("big.csv > ",sep=",",what=character(0),skip=1,n=1e6),ncol=10,byrow=TRUE) > > system("free -m") > > > > The file is approximately 9MB, but approximately 50-60MB is used to > > read it in. > > > > object.size(big1) is 56MB, or 56 bytes per string, which seems > excessive. > > > > Regards, Mike > > > > Configuration info: > > > sessionInfo() > > R version 2.5.1 (2007-06-27) > > x86_64-redhat-linux-gnu > > locale: > > C > > attached base packages: > > [1] "stats" "graphics" "grDevices" "utils" > "datasets" "methods" > > [7] "base" > > > > # uname -a > > Linux ***.com 2.6.9-023stab044.4-smp #1 SMP Thu May 24 17:20:37 MSD > > 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > ====== Quoted Text ==== > > From: Prof Brian Ripley <ripley_at_stats.ox.ac.uk> > > Date: Tue, 26 Jun 2007 17:53:28 +0100 (BST) > > > > > > > > > > The R Data Import/Export Manual points out several ways in which you > > can use read.csv more efficiently. > > > > On Tue, 26 Jun 2007, ivo welch wrote: > > > > > dear R experts: > > > > > > I am of course no R experts, but use it regularly. I thought I would > > > share some experimentation with memory use. I run a linux machine > > > with about 4GB of memory, and R 2.5.0. > > > > > > upon startup, gc() reports > > > > > > used (Mb) gc trigger (Mb) max used (Mb) > > > Ncells 268755 14.4 407500 21.8 350000 18.7 > > > Vcells 139137 1.1 786432 6.0 444750 3.4 > > > > > > This is my baseline. linux 'top' reports 48MB as baseline. This > > > includes some of my own routines that are always loaded. Good.. > > > > > > > > > Next, I created a s.csv file with 22 variables and 500,000 > > > observations, taking up an uncompressed disk space of 115MB. The > > > resulting object.size() after a read.csv() is 84,002,712 bytes (80MB). > > > > > >> s= read.csv("s.csv"); > > >> object.size(s); > > > > > > [1] 84002712 > > > > > > > > > here is where things get more interesting. after the read.csv() is > > > finished, gc() reports > > > > > > used (Mb) gc trigger (Mb) max used (Mb) > > > Ncells 270505 14.5 8349948 446.0 11268682 601.9 > > > Vcells 10639515 81.2 34345544 262.1 42834692 326.9 > > > > > > I was a big surprised by this---R had 928MB intermittent memory in > > > use. More interestingly, this is also similar to what linux 'top' > > > reports as memory use of the R process (919MB, probably 1024 vs. 1000 > > > B/MB), even after the read.csv() is finished and gc() has been run. > > > Nothing seems to have been released back to the OS. > > > > > > Now, > > > > > >> rm(s) > > >> gc() > > > used (Mb) gc trigger (Mb) max used (Mb) > > > Ncells 270541 14.5 6679958 356.8 11268755 601.9 > > > Vcells 139481 1.1 27476536 209.7 42807620 326.6 > > > > > > linux 'top' now reports 650MB of memory use (though R itself uses only > > > 15.6Mb). My guess is that It leaves the trigger memory of 567MB plus > > > the base 48MB. > > > > > > > > > There are two interesting observations for me here: first, to read a > > > .csv file, I need to have at least 10-15 times as much memory as the > > > file that I want to read---a lot more than the factor of 3-4 that I > > > had expected. The moral is that IF R can read a .csv file, one need > > > not worry too much about running into memory constraints lateron. {R > > > Developers---reducing read.csv's memory requirement a little would be > > > nice. of course, you have more than enough on your plate, already.} > > > > > > Second, memory is not returned fully to the OS. This is not > > > necessarily a bad thing, but good to know. > > > > > > Hope this helps... > > > > > > Sincerely, > > > > > > /iaw > > > > > > ______________________________________________ > > > R-help_at_stat.math.ethz.ch mailing list > > > https://stat.ethz.ch/mailman/listinfo/r-help > > > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > > > and provide commented, minimal, self-contained, reproducible code. > > > > > -- > > Brian D. Ripley, ripley_at_stats.ox.ac.uk > > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > > University of Oxford, Tel: +44 1865 272861 (self) > > 1 South Parks Road, +44 1865 272866 (PA) > > Oxford OX1 3TG, UK Fax: +44 1865 272595 > > > > ______________________________________________ > > R-help@stat.math.ethz.ch mailing list > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > > and provide commented, minimal, self-contained, reproducible code. > > > [[alternative HTML version deleted]] ______________________________________________ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.