[Bioc-devel] BiocParallel: flattening iteration

2013-11-14 Thread Michael Lawrence
Hi guys, We often need to iterate over the cartesian product of two dimensions, like sample X chromosome. This is preferable to nested iteration, which is complicated. I've been using expand.grid and bpmapply for this, but it seems like this could be made easier. Like bpmapply could gain a

Re: [Bioc-devel] BiocParallel: flattening iteration

2013-11-14 Thread Vincent Carey
Streamer package has DAGTeam/DAGParam components that I believe are relevant. An abstraction of the reduction plan for a parallelized task would seem to have a natural home in BatchJobs. On Thu, Nov 14, 2013 at 8:15 AM, Michael Lawrence lawrence.mich...@gene.com wrote: Hi guys, We often

Re: [Bioc-devel] BiocParallel: flattening iteration

2013-11-14 Thread Michel Lang
We use a design iterator in BatchExperiments::makeDesign for a cartesian product. I found a old version of designIterator (cf. https://github.com/tudo-r/BatchExperiments/blob/master/R/designs.R) w/o the optional data.frame input which is easier to read: https://gist.github.com/mllg/7469844.

Re: [Bioc-devel] BiocParallel: flattening iteration

2013-11-14 Thread Michael Lawrence
I like the general idea of having iterators; was just checking out the itertools package after not having looked at it for a while. I could see having a BiocIterators package, and a bpiterate(iterator, FUN, ..., BPPARAM). My suggestion was simpler though. Right now, bpmapply runs a single job per