RG,
Actually, SQLite provides a solution to read *.csv file directly into db.
Just for your consideration.
On 1/5/06, ronggui <[EMAIL PROTECTED]> wrote:
>
> 2006/1/6, jim holtman <[EMAIL PROTECTED]>:
> > If what you are reading in is numeric data, then it would require (807 *
> > 118519 * 8) 760MB just to store a single copy of the object -- more
> memory
> > than you have on your computer. If you were reading it in, then the
> problem
> > is the paging that was occurring.
> In fact,If I read it in 3 pieces, each is about 170M.
>
> >
> > You have to look at storing this in a database and working on a subset
> of
> > the data. Do you really need to have all 807 variables in memory at the
> > same time?
>
> Yip,I don't need all the variables.But I don't know how to get the
> necessary variables into R.
>
> At last I read the data in piece and use RSQLite package to write it
> to a database.and do then do the analysis. If i am familiar with
> database software, using database (and R) is the best choice,but
> convert the file into database format is not an easy job for me.I ask
> for help in SQLite list,but the solution is not satisfying as that
> required the knowledge about the third script language.After searching
> the internet,I get this solution:
>
> #begin
> rm(list=ls())
> f<-file("D:\wvsevs_sb_v4.csv","r")
> i <- 0
> done <- FALSE
> library(RSQLite)
> con<-dbConnect("SQLite","c:\sqlite\database.db3")
> tim1<-Sys.time()
>
> while(!done){
> i<-i+1
> tt<-readLines(f,2500)
> if (length(tt)<2500) done <- TRUE
> tt<-textConnection(tt)
> if (i==1) {
> assign("dat",read.table(tt,head=T,sep=",",quote=""));
> }
> else assign("dat",read.table(tt,head=F,sep=",",quote=""))
> close(tt)
> ifelse(dbExistsTable(con, "wvs"),dbWriteTable(con,"wvs",dat,append=T),
> dbWriteTable(con,"wvs",dat) )
> }
> close(f)
> #end
> It's not the best solution,but it works.
>
>
>
> > If you use 'scan', you could specify that you do not want some of the
> > variables read in so it might make a more reasonably sized objects.
> >
> >
> > On 1/5/06, François Pinard <[EMAIL PROTECTED]> wrote:
> > > [ronggui]
> > >
> > > >R's week when handling large data file. I has a data file : 807
> vars,
> > > >118519 obs.and its CVS format. Stata can read it in in 2 minus,but
> In
> > > >my PC,R almost can not handle. my pc's cpu 1.7G ;RAM 512M.
> > >
> > > Just (another) thought. I used to use SPSS, many, many years ago, on
> > > CDC machines, where the CPU had limited memory and no kind of paging
> > > architecture. Files did not need to be very large for being too
> large.
> > >
> > > SPSS had a feature that was then useful, about the capability of
> > > sampling a big dataset directly at file read time, quite before
> > > processing starts. Maybe something similar could help in R (that is,
> > > instead of reading the whole data in memory, _then_ sampling it.)
> > >
> > > One can read records from a file, up to a preset amount of them. If
> the
> > > file happens to contain more records than that preset number (the
> number
> > > of records in the whole file is not known beforehand), already read
> > > records may be dropped at random and replaced by other records coming
> > > from the file being read. If the random selection algorithm is
> properly
> > > chosen, it can be made so that all records in the original file have
> > > equal probability of being kept in the final subset.
> > >
> > > If such a sampling facility was built right within usual R reading
> > > routines (triggered by an extra argument, say), it could offer
> > > a compromise for processing large files, and also sometimes accelerate
> > > computations for big problems, even when memory is not at stake.
> > >
> > > --
> > > François Pinard http://pinard.progiciels-bpi.ca
> > >
> > > ______________________________________________
> > > R-help@stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> > http://www.R-project.org/posting-guide.html
> > >
> >
> >
> >
> > --
> > Jim Holtman
> > Cincinnati, OH
> > +1 513 247 0281
> >
> > What the problem you are trying to solve?
>
>
> --
> é»è£è´µ
> Deparment of Sociology
> Fudan University
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
--
WenSui Liu
(http://statcompute.blogspot.com)
Senior Decision Support Analyst
Health Policy and Clinical Effectiveness
Cincinnati Children Hospital Medical Center
[[alternative HTML version deleted]]
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
