On Tue, 17 May 2022 at 00:12, Matthew Knepley <knep...@gmail.com> wrote:
> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < > loic.goua...@polytechnique.edu> wrote: > >> Le 16/05/2022 à 21:54, Barry Smith a écrit : >> >> >> >> On May 16, 2022, at 3:50 PM, Loic Gouarin <loic.goua...@polytechnique.edu> >> wrote: >> >> Thanks Barry for your quick reply. >> Le 16/05/2022 à 21:41, Barry Smith a écrit : >> >> >> Loic, >> >> From your code it looks like you are using a DM. Is it a DMDA or a >> shell DM? If it is a DMDA then the process is intended to be pretty >> straightforward. PCTELESCOPE should create a new DMDA that has the same >> properties as the coarse grid DMDA but lives on a smaller number of ranks. >> From this you can just provide multiple levels of coarsening of the DMDA to >> produce the smaller multigrid problems. >> >> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the >> DM which is the same of the coarse grid and build all the levels as >> previously for the levels of PCTELESCOPE ? >> >> >> Yes, I think you should be able to pull out of the coarse level PC (on >> the subset of ranks) a DM that and then setup coarse levels from that. But >> you need to look at an example that that uses telescope and probably even >> the telescope code itself to see all the details. I don't know them anymore. >> >> I looked at the source code but It didn't help. I will try to explain >> more clearly my experiments to understand what I make wrong and you can >> help more easily. >> >> Hi Loic! > > Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it > creates a new DMDA and copies over all the information it can see. However, > I think you may have state that is attached to the old DMDA that is not > getting redistributed. If this is true, I can help you write a hook that > redistributes that state when TELESCOPE maps between distributions. > If you have state you need to propagate between different communicators, you will have to use the telescope type which is internally is referred to as TELESCOPE_COARSEDM To activate this type you need the option -pc_telescope_use_coarse_dm Or to call PCTelescopeSetUseCoarseDM() The assumptions of usng this options are described here https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM and source associated with the COARSEDM https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html Using TELESCOPE_COARSEDM you will have access to the hooks Matt is referring to. Cheers, Dave > Thanks, > > Matt > > >> Thanks, >> >> Loic >> >> >> Barry >> >> >> Can you let us know more details of what exactly goes wrong with >> attempts? It is likely straightforward to get things working, perhaps due >> to our unclear documentation. >> >> I essentially have wrong state or NULL matrices into the PCTELESCOPE. >> >> Loic >> >> >> Barry >> >> >> On May 16, 2022, at 3:28 PM, Loic Gouarin <loic.goua...@polytechnique.edu> >> wrote: >> >> Hello, >> >> I could have posted my message on the user list but it seems to me that >> it's more in the petsc development part. Don't hesitate to tell me if I'm >> wrong. >> I am developing a code called cafes that studies fluid-particle >> interactions in a Stokes fluid. For that, we implemented the whole solver >> in matrix free on a cartesian grid. The preconditioner is naturally a >> geometrical multigrid for the velocity. The set of matrix free for each >> level is set by hand (I worked with Matthew on this a long time ago now!). >> I would have liked to implement telescope on this part. I started it but >> I didn't get out of it. So I would like to have your help. >> What I tried to do was to describe my classical multigrid up to a level >> N1 and then take the coarse solver and apply the telescope preconditioner >> to it up to a level N2. I don't know if this is the best way but it seemed >> the most intuitive. The problem is that I could not get the coarse matrix >> because the setup is not done yet (I use a DMKSPSetComputeOperators for the >> matrices). >> >> The construction of the matrix free for each level is here: >> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L59-L88 >> >> The description of the Stokes preconditioner is described here : >> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L109-L162 >> >> I have tried multiple implementations without success. For me, the start >> is to add at line 105 something like >> KSP coarse; >> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); >> PC pc_coarse; >> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); >> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); >> >> Then, to create the several matrices of the levels of TELESCOPE, I >> thought using the same process as before with DMKSPSetComputeOperators. But >> it doesn't work... >> Could you tell me how I can make it work ? >> >> Thanks, >> Loic >> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >