On Mon, Feb 6, 2012 at 11:09 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > > Fredrik, > > This question belongs on petsc-dev at mcs.anl.gov since it involves > additions/extensions to PETSc so I am moving the discussion over to there. > > We have not done the required work to have ghosted vectors work with > CUSP yet, so this will require some additions to PETSc. We can help you > with that process but since the PETSc team does not have a CUSP person > developing PETSc full time you will need to actual contribute some code but > I'll try to guide you in the right direction. > > The first observation is that ghosted vectors in PETSc are actually > handled with largely the same code as VECMPI vectors (with just no ghost > points by default) so in theory little work needs to be done to get the > functionality you need. What makes the needed changes non-trivial is the > current interface where one calls VecCreateGhost() to create the vectors. > This is one of our "easy" interfaces and it is somewhat legacy in that > there is no way to control the types of the vectors since it creates > everything about the vector in one step. Note that we have the same > issues with regard to the pthread versions of the PETSc vectors and > ghosting. > > So before we even talk about what code to change/add we need to decide > on the interface. Presumably you want to be able to decide at runtime > whether to use regular VECMPI, VECMPICUSP or VECMPIPTHREAD in your ghosted > vectors. How do we get that information in there? An additional argument to > VecCreateGhost() (ugly?)? Options database (by calling VecSetFromOptions() > ?), other ways? So for example one could have: > > VecCreateGhost(......) > VecSetFromOptions(......) > > to set the specific type cusp or pthread? What about > > VecCreateGhost(......) > VecSetType(......,VECMPICUSP); > > which as you note doesn't currently work. Note that the PTHREAD version > needs to do its own memory allocation so essentially has to undo much of > what VecCreateGhost() already did, is that a bad thing? > > Or do we get rid of VecCreateGhost() completely and change the model to > something like > > VecCreate() > VecSetType() > VecSetGhosted() > > or > > VecCreate() > VecSetTypeFromOptions() > VecSetGhosted() > > or even > > VecCreate() > VecSetGhosted() which will just default to regular MPI ghosted. > > this model allows a clean implementation that doesn't require undoing > previously built internals. > I am for the second model, just absorbing ghosting into the current implementations. Matt > > Everyone chime in with observations so we can figure out any > refactorizations needed. > > Barry > > > > > > On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote: > > > Hi, > > > > In FEniCS, we use ghosted vectors for parallel computations, with the > functions > > > > VecCreateGhost > > VecGhostGetLocalForm > > > > As I am integrating the new GPU-based vectors and matrices in FEniCS, I > want the ghosted vectors to be of type VECMPICUSP. I have tried to do this > by calling VecSetType after creating the vector, but that makes > VecGhostGetLocalForm give an error. > > > > Is it possible to set the type to be mpicusp when using ghost vectors, > without changing much of the code? If so, how? > > > > If not, how would you recommend I proceed to work with mpicusp vectors > in this context? > > > > Thanks! > > > > -- > > Fredrik > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120206/13180f67/attachment.html>