On Nov 24, 2011, at 6:30 PM, Dmitry Karpeev wrote: > The MPICH paper anticipates O(1B) cores in an exascale machine. > The number of cores, however, needn't be the number of nodes (with one MPI > process per node), > if each node has O(1000) cores that are programmed using threads or whatever. > If we can limit the number > of nodes to O(1M) (at the expense of "fattening" the nodes to whatever crazy > core counts people envisage in > the future), then the current PetscLayout may be an acceptable price to pay. > Otherwise some sort of hierarchical PetscLayout or something entirely > different is necessary. > > In any event, this business of storing explicit ranks/offsets for the ghost > indices is sort of antithetical to the current > PetscLayout-based approach of indices having parallel semantics from the > outset. A lot of conceptual difficulties > in Sieve and MOAB (and the need for the extra objects such as Overlap and > ParallelComm (sic!)) stem from a lack of > a global address space. If we are planning to do away with PetscLayout in > its current form, I think > the whole PETSc parallelization model requires rethinking. That's definitely > not an incremental change.
Everything is on the table. I could envision a model/implementation of PETSc that has no concept of global numbering and always works with rank,local indexing. Being able to make a user friendly system with any model/implementation, of course, is crucial to whether the proposed system makes any sense. Barry > > Dmitry. > > On Thu, Nov 24, 2011 at 6:17 PM, Dmitry Karpeev <karpeev at mcs.anl.gov> > wrote: > So then are you planning to re-engineer PetscLayout? > Or is it acceptable to have a few PetscLayouts, in their current form, lying > around? > > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev at mcs.anl.gov> wrote: > Is O(commSize) storage considered unscalable in this model? > > I would like to avoid it where possible. If we have million-way parallelism > at the network level (e.g. one million MPI processes), then we have to > justify the memory usage. One 64-bit integer per process is 8MB. If that is > needed once for the entire application, it is probably not an issue, but if > it's needed by every object, then it could add up to a substantial amount > (considering the low-memory estimates). > > The MPICH guys are thinking about this: > > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf > >
