[R] Manipulate single line in textfile
Hello all, Is it possible to modify a single line in a textfile? I know it is possible to load the whole text file, do the change, and save this as a new file. However, this is not practical in my case, because the document is huge and cannot be fully loaded in R. Any idea? Best, Guillaume __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] How to separate huge dataset into chunks
Hello Thomas,
Thanks for your help!
Sadly your code does not work for the last chunk, because its length is
shorter than nrows.
I tried
try(chunk<-read.table(conn, nrows=1,col.names=nms), silent=TRUE)
but it gives me an error (go figure!)
Best,
Guillaume
Quoting Thomas Lumley :
On Tue, 24 Mar 2009, Guillaume Filteau wrote:
Hello all,
Im trying to take a huge dataset (1.5 GB) and separate it into
smaller chunks with R.
So far I had nothing but problems.
I cannot load the whole dataset in R due to memory problems. So, I
instead try to load a few (10) lines at a time (with read.table).
However, R kept crashing (with no error message) at about the
680 line. This is extremely frustrating.
To try to fix this, I used connections with read.table. However, I
now get a cryptic error telling me no lines available in input.
Is there any way to make this work?
There might be an error in line 42 of your script. Or somewhere else.
The error message is cryptically saying that there were no lines of
text available in the input connection, so presumably the connection
wasn't pointed at your file correctly.
It's hard to guess without seeing what you are doing, but
conn <- file("mybigfile", open="r")
chunk<- read.table(conn, header=TRUE, nrows=1)
nms <- names(chunk)
while(length(chunk)==1){
chunk<-read.table(conn, nrows=1,col.names=nms)
## do something to the chunk
}
close(conn)
should work. This sort of thing certainly does work routinely.
It's probably not worth reading 100,000 lines at a time unless your
computer has a lot of memory. Reducing the chunk size to 10,000
shouldn't introduce much extra overhead and may well increase the
speed by reducing memory use.
-thomas
Thomas Lumley Assoc. Professor, Biostatistics
tlum...@u.washington.eduUniversity of Washington, Seattle
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] How to separate huge dataset into chunks
Hello all, Im trying to take a huge dataset (1.5 GB) and separate it into smaller chunks with R. So far I had nothing but problems. I cannot load the whole dataset in R due to memory problems. So, I instead try to load a few (10) lines at a time (with read.table). However, R kept crashing (with no error message) at about the 680 line. This is extremely frustrating. To try to fix this, I used connections with read.table. However, I now get a cryptic error telling me no lines available in input. Is there any way to make this work? Best, Guillaume __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] How do I add a variable to a text file?
Hello all, I have a 2.0 GB dataset that I can't load into R, due to memory issues. The dataset itself is in a tab-delimited .txt file with 25 variables. I have a variable I'd like to add to the dataset. How do I do this? Best, Guillaume __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Symbols to use in text
Hello all, I'm trying to use text symbols in R, such are greek letters and mathematical operators. Do you know where I can find a list telling me how to use all those symbols? I'd like to have them displayed in my graphs. Best, Guillaume __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Matrix starting at [0,0] instead of [1,1]?
Hello all, When I create a matrix, is there a way to make it start at [0,0], instead of [1,1]? That way, a 2x2 matrix would go from [0,0] to [1,1], instead of [1,1] to [2,2]. Best, Guillaume __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
