Junchao, Thanks for checking this.
Could you use PetscRegisterFinalize()? Barry > On Aug 14, 2021, at 10:46 PM, Junchao Zhang <junchao.zh...@gmail.com> wrote: > > > > On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > User visible communicators generally do not have a keyval attached. Rather > the keyval is attached to the inner communicator; because we don't want both > PETSc and the user doing MPI operations on the same communicator (to prevent > mixing up tags). > > I think PetscShmCommGet() is wrong. I think it should not call > MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should > call PetscCommDuplicate() and use that communicator to stash the pshmcomm; > then you would not have the problem you are having. > Barry, I think it over and find the problem is: if I PetscCommDuplicate() > the outer comm in PetscShmCommGet(), then I can not find a place to destroy > the inner communicator (note usually petsc inner communciators are destroyed > with petsc objects. Doing what you said breaks the rule) > > PetscShmCommGet() was designed to help doing OpenMP multithreading on a > communicator that some petsc objects live in. So requiring the input > communicator to be petsc comm is not totally nonsense. > > I tried another approach: Don't check whether the input comm in > PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We > instead check its Petsc_ShmComm_keyval. If it does not have one, we just > create one for it (along with a new shared memory communicator) > With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...). The > problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted inside > MPI_Finalize(). PETSc -malloc_dump complains of unfreed memory (since I > used PetscMalloc inside PetscShmCommGet). I could > bypass PetscMalloc and directly use malloc() to avoid this situation. Is it > worthy? > > > > Barry > > > > > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pie...@joliv.et > > <mailto:pie...@joliv.et>> wrote: > > > > Hello, > > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc > > communicator, i.e., has no Petsc_Counter_keyval attached? > > ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr); > > yields > > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc communicator > > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60 > > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340 > > > > Thanks, > > Pierre >