Dear Anton, 4Mb is not a lot of data. A Gb still wouldn't be that troublesome in a flat file. Your data can be migrated to a relational database at a future point.
Sincerely, KeithC. -----Original Message----- From: Anton du Toit [mailto:atdutoitrh...@gmail.com] Sent: Saturday, February 13, 2010 3:54 AM To: r-help Subject: Re: [R] Hierarchical data sets: which software to use? Hi Douglas, Thanks for your helpful response. I've commented on some of the points you raised below: > Although this may not be helpful for your immediate goal, storing and manipulating data of this size and complexity (and, I expect, cost for collection) really calls for tools like relational databases. A single flat file of 2500 variables by 1500 cases is almost never the best way to organize such data. A normalized representation as a collection of interlinked tables in a relational data base is much more effective and less error prone. The widespread use of spreadsheets or SPSS data sets or SAS data sets which encourage the "single table with a gargantuan number of columns, most of which are missing data in most cases" approach to organization of longitudinal data is regrettable. I'm both relieved and daunted by this. Daunted because it means I'll need to learn another package (probably postGreSQL or MySQL?), but relieved because constructing a 2500 by 1500 file seemed intuitively wrong, as well as introducing the possibility of errors unnecessarily--surely it makes more sense to leave the data as is. As far as immediate goals go--I am at the beginning of a thesis, and I have more research planned after that, so I want to get things right from the start. > For later analysis in R it is better to start with "long" form of the data, as opposed to the "wide" form, even if it means repeating demographic information over several occasions. Using a relational database allows for a long view to be generated without the possibility of inconsistency in the demographics. I am using the descriptions "long" and "wide" in the sense that they are used in the reshape help page. See ?reshape > in R. The long view is also called the subject/occasion view in the sense that each row corresponds to one subject on one occasion. > Robert Gentleman's book "R Programming for Bioinformatics" provides background on linking R to relational databases. Thanks--I'll look this one up. > As I said at the beginning, you may not want to undertake the necessary study and effort to reorganize your data for this specific project but if you do this a lot you may want to consider it. As above: a stitch in time, I suppose. Thanks again. Anton On Sat, Feb 6, 2010 at 3:22 AM, Douglas Bates <ba...@stat.wisc.edu> wrote: > On Sun, Jan 31, 2010 at 10:24 PM, Anton du Toit <atdutoitrh...@gmail.com> wrote: >> Dear R-helpers, >> >> Im writing for advice on whether I should use R or a different >> package or language. Ive looked through the R-help archives, some >> manuals, and some other sites as well, and I havent done too well >> finding relevant info, hence my question here. >> >> Im working with hierarchical data (in SPSS lingo). That is, for each >> case >> (person) I read in three types of (medical) record: >> >> 1. demographic data: name, age, sex, address, etc >> >> 2. admissions data: this generally repeats, so I will have 20 or so >> variables relating to their first hospital admission, then the same >> 20 again for their second admission, and so on >> >> 3. collections data, about 100 variables containing the results of >> a battery of standard tests. These are administered at intervals and >> so this is repeating data as well. >> >> The number of repetitions varies between cases, so in its one case >> per line format the data is non-rectangular. >> >> At present I have shoehorned all of this into SPSS, with each case on >> one line. My test database has 2,500 variables and 1,500 cases (or >> persons), and in SPSSs *.SAV format is ~4MB. The one I finally work >> with will be larger again, though likely within one order of >> magnitude. Down the track, funding permitting, I hope to be working with tens of thousands of cases. > > Although this may not be helpful for your immediate goal, storing and > manipulating data of this size and complexity (and, I expect, cost for > collection) really calls for tools like relational databases. A > single flat file of 2500 variables by 1500 cases is almost never the > best way to organize such data. A normalized representation as a > collection of interlinked tables in a relational data base is much > more effective and less error prone. The widespread use of > spreadsheets or SPSS data sets or SAS data sets which encourage the > "single table with a gargantuan number of columns, most of which are > missing data in most cases" approach to organization of longitudinal > data is regrettable. > > For later analysis in R it is better to start with "long" form of the > data, as opposed to the "wide" form, even if it means repeating > demographic information over several occasions. Using a relational > database allows for a long view to be generated without the > possibility of inconsistency in the demographics. I am using the > descriptions "long" and "wide" in the sense that they are used in the > reshape help page. See > > ?reshape > > in R. The long view is also called the subject/occasion view in the > sense that each row corresponds to one subject on one occasion. > > Robert Gentleman's book "R Programming for Bioinformatics" provides > background on linking R to relational databases. > > > As I said at the beginning, you may not want to undertake the > necessary study and effort to reorganize your data for this specific > project but if you do this a lot you may want to consider it. > >> I am wondering if I should keep using SPSS, or try something else. >> >> The types of analysis Ill typically will have to do will involve >> comparing measurements at different times, e.g. before/ after >> treatment. Ill also need to compare groups of people, e.g. treatment >> / no treatment. Regression and factor analyses will doubtless come into it at some point too. >> >> So: >> >> 1. should I use R or try something else? >> >> 2. can anyone advise me on using R with the type of data Ive described? >> >> >> Many thanks, >> >> Anton du Toit ______________________________________________ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.