>> How many vectors are you storing in this application? > > Quite a lot. (-: > > (About 8 systems, most of which are transient, and some of which have > additional vectors.) > >> Redundant vector storage being the limiting factor in scaling at this scale >> surprises me... > > Roy was also surprised when I told him this the first time, but it's > really true. Well, at least, for 8 CPUs, the per-CPU memory usage > drops a lot when using ghosted vectors. That's why I pushed their > implementation. From 8 to 24 CPUs (that's running right now), > however, the per-CPU memory usage does actually not decrease > essentially any more.
I have to admit I have not been following this thread as closely as I would have liked to (or should have)... As for the performance implications, I think we can get nearly all of it back by optimizing the underlying PetscVector (but actually do it in the NumericVector base class) to bypass the PETSc API whenever possible. Specifically, we would use the VecCreateGhostWithArray (sp?) method to provide the storage buffer and then access it directly whenever possible. I need to look at the trilinos documentation in this regard too, though, as I'd like to kill two birds with one stone so-to-speak by performing the optmization in the base class in such a way that both derived classes can benefit. -Ben ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Libmesh-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-devel
