If this is an option for you: An xml database can handle (very) huge xml files and let you query nodes very efficiently. Then, you could query the xml databse from R (using REST) to do your statistics.
There are some open source xquery/xml databases available. 2012/8/11 Frederic Fournier <frederic.bioi...@gmail.com> > Hello everyone, > > I would like to parse very large xml files from MS/MS experiments and > create R objects from their content. (By very large, I mean going up to > 5-10Gb, although I am using a 'small' 40M file to test my code.) > > My first attempt at parsing the 40M file, using the XML package, took more > than 2200 seconds and left me quite disappointed. > I managed to cut that down to around 40 seconds by: > -using the 'useInternalNodes' option of the XML package when parsing > the xml tree; > -vectorizing the parsing (i.e., replacing loops like "for(node in > group.of.nodes) {...}" by "sapply(group.of.node, function(node){...}") > I gained another 5 seconds by making small changes to the functions used > (like replacing 'getNodeset' by 'xmlElementsByTagName' when I don't need to > navigate to the children nodes). > Now I am blocked at around 35 seconds and I would still like to cut this > time by a 5x, but I have no clue what to do to achieve this gain. I'll try > to expose as briefly as possible the relevant structure of the xml file I > am parsing, the structure of the R object I want to create, and the type of > functions I am using to do it. I hope that one of you will be able to point > me towards a better and quicker way of doing the parsing! > > > Here is the (simplified) structure of the relevant nodes of the xml file: > > <model> (many many nodes) > <protein> (a couple of proteins per model node) > <peptide> (1 per protein node) > <domain> (1 or more per peptide node) > <aa> (0 or more per domain node) > </aa> > </domain> > </peptide> > </protein> > </model> > > Here is the basic structure of the R object that I want to create: > > 'result' object that contains: > -various attributes > -a list of 'protein' objects, each of which containing: > -various attributes > -a list of 'peptide' objects, each of which containing: > -various attributes > -a list of 'aa' objects, each of which consisting of a couple of > attributes. > > Here is the basic structure of the code: > > xml.doc <- xmlTreeParse("file", getDTD=FALSE, useInternalNodes=TRUE) > result <- new('S4_result_class') > result@proteins <- xpathApply(xml.doc, "//model/protein", > function(protein.node) { > protein <- new('S4_protein_class') > ## fill in a couple of attributes of the protein object using xmlValue > and xmlAttrs(protein.node) > protein@peptides <- xpathApply(protein.node, "./peptide", > function(peptide.node) { > peptide <- new('S4_peptide_class') > ## fill in a couple of attributes of the peptide object using xmlValue > and xmlAttrs(peptide.node) > peptide@aas <- sapply(xmlElementsByTagName(peptide.node, name="aa"), > function(aa.node) { > aa <- new('S4_aa_class') > ## fill in a couple of attributes of the 'aa' object using xmlValue > and xmlAttrs(aa.node) > }) > }) > }) > free(xml.doc) > > > Does anyone know a better and quicker way of doing this? > > Sorry for the very long message and thank you very much for your time and > help! > > Frederic > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.