Checking back. What does not work? Thanks,
Matt On Tue, Jan 24, 2023 at 11:26 AM Matthew Knepley <knep...@gmail.com> wrote: > On Tue, Jan 24, 2023 at 10:39 AM Berend van Wachem < > berend.vanwac...@ovgu.de> wrote: > >> Dear Matt, >> >> I have been working on this now with Petsc-3.18.3 >> >> 1) I can confirm that enforcing periodicity works for a single core >> simulation. >> >> 2) However, when using multiple cores, the code still hangs. Is there >> something I should do to fix this? Or should this be fixed in the next >> Petsc version? >> > > Dang dang dang. I forgot to merge this fix. Thanks for reminding me. It is > now here: > > https://gitlab.com/petsc/petsc/-/merge_requests/6001 > > >> 3) This is strange, as it works fine for me. >> > > Will try again with current main. > > Thanks > > Matt > > >> Thanks, best, Berend. >> >> >> On 12/15/22 18:56, Matthew Knepley wrote: >> > On Wed, Dec 14, 2022 at 3:58 AM Berend van Wachem >> > <berend.vanwac...@ovgu.de <mailto:berend.vanwac...@ovgu.de>> wrote: >> > >> > >> > Dear PETSc team and users, >> > >> > I have asked a few times about this before, but we haven't really >> > gotten >> > this to work yet. >> > >> > In our code, we use the DMPlex framework and are also interested in >> > periodic geometries. >> > >> > As our simulations typically require many time-steps, we would like >> to >> > be able to save the DM to file and to read it again to resume the >> > simulation (a restart). >> > >> > Although this works for a non-periodic DM, we haven't been able to >> get >> > this to work for a periodic one. To illustrate this, I have made a >> > working example, consisting of 2 files, createandwrite.c and >> > readandcreate.c. I have attached these 2 working examples. We are >> using >> > Petsc-3.18.2. >> > >> > In the first file (createandwrite.c) a DMPlex is created and >> written to >> > a file. Periodicity is activated on lines 52-55 of the code. >> > >> > In the second file (readandcreate.c) a DMPlex is read from the file. >> > When a periodic DM is read, this does not work. Also, trying to >> > 'enforce' periodicity, lines 55 - 66, does not work if the number of >> > processes is larger than 1 - the code "hangs" without producing an >> > error. >> > >> > Could you indicate what I am missing? I have really tried many >> > different >> > options, without finding a solution. >> > >> > >> > Hi Berend, >> > >> > There are several problems. I will eventually fix all of them, but I >> > think we can get this working quickly. >> > >> > 1) Periodicity information is not saved. I will fix this, but forcing >> it >> > should work. >> > >> > 2) You were getting a hang because the blocksize on the local >> > coordinates was not set correctly after loading >> > since the vector had zero length. This does not happen in any >> test >> > because HDF5 loads a global vector, but >> > most other things create local coordinates. I have a fix for >> this, >> > which I will get in an MR, Also, I moved DMLocalizeCoordinates() >> > after distribution, since this is where it belongs. >> > >> > knepley/fix-plex-periodic-faces *$:/PETSc3/petsc/petsc-dev$ git diff >> > diff --git a/src/dm/interface/dmcoordinates.c >> > b/src/dm/interface/dmcoordinates.c >> > index a922348f95b..6437e9f7259 100644 >> > --- a/src/dm/interface/dmcoordinates.c >> > +++ b/src/dm/interface/dmcoordinates.c >> > @@ -551,10 +551,14 @@ PetscErrorCode DMGetCoordinatesLocalSetUp(DM dm) >> > PetscFunctionBegin; >> > PetscValidHeaderSpecific(dm, DM_CLASSID, 1); >> > if (!dm->coordinates[0].xl && dm->coordinates[0].x) { >> > - DM cdm = NULL; >> > + DM cdm = NULL; >> > + PetscInt bs; >> > >> > PetscCall(DMGetCoordinateDM(dm, &cdm)); >> > PetscCall(DMCreateLocalVector(cdm, &dm->coordinates[0].xl)); >> > + // If the size of the vector is 0, it will not get the right block >> size >> > + PetscCall(VecGetBlockSize(dm->coordinates[0].x, &bs)); >> > + PetscCall(VecSetBlockSize(dm->coordinates[0].xl, bs)); >> > PetscCall(PetscObjectSetName((PetscObject)dm->coordinates[0].xl, >> > "coordinates")); >> > PetscCall(DMGlobalToLocalBegin(cdm, dm->coordinates[0].x, >> > INSERT_VALUES, dm->coordinates[0].xl)); >> > PetscCall(DMGlobalToLocalEnd(cdm, dm->coordinates[0].x, >> > INSERT_VALUES, dm->coordinates[0].xl)); >> > >> > 3) If I comment out forcing the periodicity, your example does not >> run >> > for me. I will try to figure it out >> > >> > [0]PETSC ERROR: --------------------- Error Message >> > -------------------------------------------------------------- >> > [0]PETSC ERROR: Nonconforming object sizes >> > [0]PETSC ERROR: SF roots 4400 < pEnd 6000 >> > [1]PETSC ERROR: --------------------- Error Message >> > -------------------------------------------------------------- >> > [0]PETSC ERROR: WARNING! There are option(s) set that were not used! >> > Could be the program crashed before they were used or a spelling >> > mistake, etc! >> > [1]PETSC ERROR: Nonconforming object sizes >> > [0]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) >> > source: command line >> > [1]PETSC ERROR: SF roots 4400 < pEnd 6000 >> > [0]PETSC ERROR: See https://petsc.org/release/faq/ >> > <https://petsc.org/release/faq/> for trouble shooting. >> > [0]PETSC ERROR: Petsc Development GIT revision: >> v3.18.1-494-g16200351da0 >> > GIT Date: 2022-12-12 23:42:20 +0000 >> > [1]PETSC ERROR: WARNING! There are option(s) set that were not used! >> > Could be the program crashed before they were used or a spelling >> > mistake, etc! >> > [1]PETSC ERROR: Option left: name:-start_in_debugger_no (no value) >> > source: command line >> > [0]PETSC ERROR: ./readandcreate on a arch-master-debug named >> > MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by >> knepley >> > Thu Dec 15 12:50:26 2022 >> > [1]PETSC ERROR: See https://petsc.org/release/faq/ >> > <https://petsc.org/release/faq/> for trouble shooting. >> > [0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug >> > --download-bamg --download-bison --download-chaco --download-ctetgen >> > --download-egads --download-eigen --download-exodusii --download-fftw >> > --download-hpddm --download-ks --download-libceed --download-libpng >> > --download-metis --download-ml --download-mumps --download-muparser >> > --download-netcdf --download-opencascade --download-p4est >> > --download-parmetis --download-pnetcdf --download-pragmatic >> > --download-ptscotch --download-scalapack --download-slepc >> > --download-suitesparse --download-superlu_dist --download-tetgen >> > --download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake >> > --with-ctest-exec=/PETSc3/petsc/apple/bin/ctest >> > --with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple >> > --with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib >> > [1]PETSC ERROR: Petsc Development GIT revision: >> v3.18.1-494-g16200351da0 >> > GIT Date: 2022-12-12 23:42:20 +0000 >> > [0]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at >> > /PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322 >> > [1]PETSC ERROR: ./readandcreate on a arch-master-debug named >> > MacBook-Pro.cable.rcn.com <http://MacBook-Pro.cable.rcn.com> by >> knepley >> > Thu Dec 15 12:50:26 2022 >> > [0]PETSC ERROR: #2 DMGetGlobalSection() at >> > /PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527 >> > [1]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug >> > --download-bamg --download-bison --download-chaco --download-ctetgen >> > --download-egads --download-eigen --download-exodusii --download-fftw >> > --download-hpddm --download-ks --download-libceed --download-libpng >> > --download-metis --download-ml --download-mumps --download-muparser >> > --download-netcdf --download-opencascade --download-p4est >> > --download-parmetis --download-pnetcdf --download-pragmatic >> > --download-ptscotch --download-scalapack --download-slepc >> > --download-suitesparse --download-superlu_dist --download-tetgen >> > --download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake >> > --with-ctest-exec=/PETSc3/petsc/apple/bin/ctest >> > --with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple >> > --with-petsc4py=1 --with-shared-libraries --with-slepc --with-zlib >> > [0]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at >> > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750 >> > [1]PETSC ERROR: #1 PetscSectionCreateGlobalSection() at >> > /PETSc3/petsc/petsc-dev/src/vec/is/section/interface/section.c:1322 >> > [0]PETSC ERROR: #4 DMPlexSectionLoad() at >> > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364 >> > [1]PETSC ERROR: #2 DMGetGlobalSection() at >> > /PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:4527 >> > [0]PETSC ERROR: #5 main() at >> > /Users/knepley/Downloads/tmp/Berend/readandcreate.c:85 >> > [1]PETSC ERROR: #3 DMPlexSectionLoad_HDF5_Internal() at >> > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:2750 >> > [0]PETSC ERROR: PETSc Option Table entries: >> > [0]PETSC ERROR: -malloc_debug (source: environment) >> > [1]PETSC ERROR: #4 DMPlexSectionLoad() at >> > /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:2364 >> > [1]PETSC ERROR: #5 main() at >> > /Users/knepley/Downloads/tmp/Berend/readandcreate.c:85 >> > [0]PETSC ERROR: -start_in_debugger_no (source: command line) >> > [1]PETSC ERROR: PETSc Option Table entries: >> > [0]PETSC ERROR: ----------------End of Error Message -------send entire >> > error message to petsc-ma...@mcs.anl.gov---------- >> > [1]PETSC ERROR: -malloc_debug (source: environment) >> > application called MPI_Abort(MPI_COMM_SELF, 60) - process 0 >> > [1]PETSC ERROR: -start_in_debugger_no (source: command line) >> > [1]PETSC ERROR: ----------------End of Error Message -------send entire >> > error message to petsc-ma...@mcs.anl.gov---------- >> > application called MPI_Abort(MPI_COMM_SELF, 60) - process 0 >> > 4) We now have parallel HDF5 loading, so you should not have to >> manually >> > distribute. I will change your example to use it >> > and send it back when I am done. >> > >> > Thanks! >> > >> > Matt >> > >> > Many thanks and kind regards, >> > Berend. >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> > experiments is infinitely more interesting than any results to which >> > their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ < >> http://www.cse.buffalo.edu/~knepley/> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>