On Tue, Oct 11, 2022 at 6:04 PM Mike Michell <mi.mike1...@gmail.com> wrote:
> Thank you for the reply and checking. > Indeed, it seems that local-to-local halo is still not implemented for > DMPlex. But I believe this is a very required feature for large 3D > simulations with DMPlex. > It is possible that it will make a difference. An explicit code, running completely from local vectors, which assembles its own residuals without mapping to global vectors, could probably realize some gain. It would be interesting to see the numbers. > Would you mind if I ask for any estimated timeline to fix this issue and > put it on the official version of PETSc? If I remember correctly, we had a > similar discussion a few months ago. > I cannot do it until November since we have a big review coming up. Thanks, Matt > Thanks, > Mike > > On Mon, Oct 10, 2022 at 10:41 PM Mike Michell <mi.mike1...@gmail.com> >> wrote: >> >>> Hi, I was wondering if there is any comment on the example file that I >>> can refer to. >>> >> >> I see the problem. Local2Local is not implemented for Plex. I thought we >> had this automated, but it was only >> coded for DMDA. It is a fairly mechanical transformation of the >> Global2Local, just remapping indices, but it >> will take some time since there is a lot of backlog this semester. >> >> I have fixed the error message so now it is obvious what the problem is. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Mike >>> >>> >>>> Thank you for the reply. >>>> Sure, a short example code is attached here with a square box mesh and >>>> a run script. >>>> Inside the source, you may find two versions of halo exchange; one is >>>> for local to global (Version-1) and another one is for local to local >>>> (Version-2), which is not working in my case. >>>> In the output.vtu, you will see the halo exchanged vector resolved to >>>> each vertex with (myrank + 1), so if the code is running with 2procs, at >>>> the parallel boundary, you will see 3. In this example, there is no ghost >>>> layer. >>>> >>>> Thanks, >>>> Mike >>>> >>>> >>>>> On Sat, Oct 1, 2022 at 8:51 PM Mike Michell <mi.mike1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thank you for the reply. There is that file in >>>>>> src/dm/interface/ftn-auto/ for me, instead of the path you mentioned. >>>>>> >>>>>> After "make allfortranstubs" was done and, PETSc reconfigured and >>>>>> reinstalled. >>>>>> >>>>>> However, I still have the same problem at the line in which >>>>>> DMLocalToLocalBegin() is used. What I am doing to setup halo exchange is >>>>>> as >>>>>> follows; >>>>>> - declare DMPlex >>>>>> - PetscSectionCreate() >>>>>> - PetscSectionSetNumFields() >>>>>> - PetscSectionSetFieldComponents() >>>>>> - PetscSectionSetChart() >>>>>> - do loop over dofs: PetscSectionSetDof() and >>>>>> PetscSectionSetFieldDof() >>>>>> - PetscSectionSetUp() >>>>>> - DMSetLocalSection() >>>>>> - PetscSectionDestroy() >>>>>> - DMGetSectionSF() >>>>>> - PetscSFSetUp() >>>>>> >>>>>> Then, the halo exchange is performed as follows; >>>>>> - DMGetLocalVector() >>>>>> - Fill the local vector >>>>>> - DMLocalToLocalBegin() --(*) >>>>>> - DMLocalToLocalEnd() >>>>>> - DMRestoreLocalVector() >>>>>> >>>>>> Then, the code crashes at (*). >>>>>> >>>>> >>>>> Can you send something I can run? Then I will find the problem and fix >>>>> it. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Previously(at the time PETSc did not support LocalToLocal for DMPlex >>>>>> in Fortran), the part above, "DMLocalToLocalBegin() and >>>>>> DMLocalToLocalEnd()", consisted of; >>>>>> - DMLocalToGlobalBegin() >>>>>> - DMLocalToGlobalEnd() >>>>>> - DMGlobalToLocalBegin() >>>>>> - DMGlobalToLocalEnd() >>>>>> and it worked okay. >>>>>> >>>>>> I am unclear which part is causing the problem. Shall I define the >>>>>> PetscSection and PetscSF in different ways in case of Local to Local Halo >>>>>> exchange? >>>>>> Any comment will be appreciated. >>>>>> >>>>>> Thanks, >>>>>> Mike >>>>>> >>>>>> >>>>>> >>>>>>> On Fri, Sep 30, 2022 at 4:14 PM Mike Michell <mi.mike1...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> As a follow-up to this email thread, >>>>>>>> https://www.mail-archive.com/petsc-users@mcs.anl.gov/msg44070.html >>>>>>>> >>>>>>>> Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available >>>>>>>> for DMPlex with Fortran on the latest version of PETSc (3.17.99 from >>>>>>>> GitLab)? Matt commented that the Fortran bindings were updated so that >>>>>>>> those functions must be available in the latest version of PETSc, >>>>>>>> however, >>>>>>>> it seems still they are not working from my test with DMPlex in >>>>>>>> Fortran. >>>>>>>> Can anyone provide some comments? Probably I am missing some mandatory >>>>>>>> header file? Currently, I have headers; >>>>>>>> >>>>>>>> #include "petsc/finclude/petscvec.h" >>>>>>>> #include "petsc/finclude/petscdmplex.h" >>>>>>>> #include "petsc/finclude/petscdmlabel.h" >>>>>>>> #include "petsc/finclude/petscdm.h" >>>>>>>> >>>>>>> >>>>>>> The source for these functions is in >>>>>>> >>>>>>> src/dm/ftn-auto/dmf.c >>>>>>> >>>>>>> Is it there for you? If not, you can run >>>>>>> >>>>>>> make allfortranstubs >>>>>>> >>>>>>> Fortran functions are not declared, so the header should not matter >>>>>>> for compilation, just the libraries for linking. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> Mike >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>