Re: [petsc-dev] Telescope usage
Le 17/05/2022 à 01:43, Dave May a écrit : On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: On Mon, May 16, 2022 at 9:59 AM Loic Gouarin wrote: Le 16/05/2022 à 21:54, Barry Smith a écrit : On May 16, 2022, at 3:50 PM, Loic Gouarin wrote: Thanks Barry for your quick reply. Le 16/05/2022 à 21:41, Barry Smith a écrit : Loic, From your code it looks like you are using a DM. Is it a DMDA or a shell DM? If it is a DMDA then the process is intended to be pretty straightforward. PCTELESCOPE should create a new DMDA that has the same properties as the coarse grid DMDA but lives on a smaller number of ranks. From this you can just provide multiple levels of coarsening of the DMDA to produce the smaller multigrid problems. It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the DM which is the same of the coarse grid and build all the levels as previously for the levels of PCTELESCOPE ? Yes, I think you should be able to pull out of the coarse level PC (on the subset of ranks) a DM that and then setup coarse levels from that. But you need to look at an example that that uses telescope and probably even the telescope code itself to see all the details. I don't know them anymore. I looked at the source code but It didn't help. I will try to explain more clearly my experiments to understand what I make wrong and you can help more easily. Hi Loic! Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it creates a new DMDA and copies over all the information it can see. However, I think you may have state that is attached to the old DMDA that is not getting redistributed. If this is true, I can help you write a hook that redistributes that state when TELESCOPE maps between distributions. If you have state you need to propagate between different communicators, you will have to use the telescope type which is internally is referred to as TELESCOPE_COARSEDM To activate this type you need the option -pc_telescope_use_coarse_dm Or to call PCTelescopeSetUseCoarseDM() The assumptions of usng this options are described here https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM and source associated with the COARSEDM https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html Using TELESCOPE_COARSEDM you will have access to the hooks Matt is referring to. Thanks Matthew and Dave. I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero instances of a coarse DM were found". And indeed, my coarse solver only has the DM on the coarse mesh and not on the fine level with a coarse DM atttached. I think it's because I use DMKSPSetComputeOperators to create the KSP on each level. So for now, I just added KSP coarse; ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); PC pc_coarse; ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); ierr=PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); but it's not enough. Cheers, Loic Cheers, Dave Thanks, Matt Thanks, Loic Barry Can you let us know more details of what exactly goes wrong with attempts? It is likely straightforward to get things working, perhaps due to our unclear documentation. I essentially have wrong state or NULL matrices into the PCTELESCOPE. Loic Barry On May 16, 2022, at 3:28 PM, Loic Gouarin wrote: Hello, I could have posted my message on the user list but it seems to me that it's more in the petsc development part. Don't hesitate to tell me if I'm wrong. I am developing a code called cafes that studies fluid-particle interactions in a Stokes fluid. For that, we implemented the whole solver in matrix free on a cartesian grid. The preconditioner is naturally a geometrical multigrid for the velocity. The set of matrix free for each level is set by hand (I worked with Matthew on this a long time ago now!). I would have liked to implement telescope on this part. I started it but I didn't get out of it. So I would like to have your help. What I tried to do was to describe my classical multigrid up to a level N1 and then take the coarse solver and apply the telescope preconditioner to it up to a level N2. I don't know if this is the best way but it seemed the most intuitive. The problem is that I could not get the coarse matrix because the setup is not done yet (I use a DMKSPSetComputeOperators for the matrices). The construct
Re: [petsc-dev] Telescope usage
Hi Loic, Can you confirm if your problem has stored state which needs to be repartitioned? Thanks, Dave On Tue 17. May 2022 at 09:07, Loic Gouarin wrote: > > Le 17/05/2022 à 01:43, Dave May a écrit : > > > > On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: > >> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < >> loic.goua...@polytechnique.edu> wrote: >> >>> Le 16/05/2022 à 21:54, Barry Smith a écrit : >>> >>> >>> >>> On May 16, 2022, at 3:50 PM, Loic Gouarin < >>> loic.goua...@polytechnique.edu> wrote: >>> >>> Thanks Barry for your quick reply. >>> Le 16/05/2022 à 21:41, Barry Smith a écrit : >>> >>> >>> Loic, >>> >>> From your code it looks like you are using a DM. Is it a DMDA or a >>> shell DM? If it is a DMDA then the process is intended to be pretty >>> straightforward. PCTELESCOPE should create a new DMDA that has the same >>> properties as the coarse grid DMDA but lives on a smaller number of ranks. >>> From this you can just provide multiple levels of coarsening of the DMDA to >>> produce the smaller multigrid problems. >>> >>> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the >>> DM which is the same of the coarse grid and build all the levels as >>> previously for the levels of PCTELESCOPE ? >>> >>> >>> Yes, I think you should be able to pull out of the coarse level PC (on >>> the subset of ranks) a DM that and then setup coarse levels from that. But >>> you need to look at an example that that uses telescope and probably even >>> the telescope code itself to see all the details. I don't know them anymore. >>> >>> I looked at the source code but It didn't help. I will try to explain >>> more clearly my experiments to understand what I make wrong and you can >>> help more easily. >>> >> Hi Loic! >> >> Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it >> creates a new DMDA and copies over all the information it can see. However, >> I think you may have state that is attached to the old DMDA that is not >> getting redistributed. If this is true, I can help you write a hook that >> redistributes that state when TELESCOPE maps between distributions. >> > > If you have state you need to propagate between different communicators, > you will have to use the telescope type which is internally is referred to > as > > TELESCOPE_COARSEDM > > > To activate this type you need the option > > -pc_telescope_use_coarse_dm > > Or to call PCTelescopeSetUseCoarseDM() > The assumptions of usng this options are described here > https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM > > and source associated with the COARSEDM > https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html > > Using TELESCOPE_COARSEDM you will have access to the hooks Matt is > referring to. > > Thanks Matthew and Dave. > > I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero > instances of a coarse DM were found". And indeed, my coarse solver only has > the DM on the coarse mesh and not on the fine level with a coarse DM > atttached. I think it's because I use DMKSPSetComputeOperators to create > the KSP on each level. > > So for now, I just added > KSP coarse; > ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); > PC pc_coarse; > ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); > ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); > ierr = PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); > > but it's not enough. > > Cheers, > > Loic > > > Cheers, > Dave > > > >>Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Loic >>> >>> >>> Barry >>> >>> >>>Can you let us know more details of what exactly goes wrong with >>> attempts? It is likely straightforward to get things working, perhaps due >>> to our unclear documentation. >>> >>> I essentially have wrong state or NULL matrices into the PCTELESCOPE. >>> >>> Loic >>> >>> >>> Barry >>> >>> >>> On May 16, 2022, at 3:28 PM, Loic Gouarin < >>> loic.goua...@polytechnique.edu> wrote: >>> >>> Hello, >>> >>> I could have posted my message on the user list but it seems to me that >>> it's more in the petsc development part. Don't hesitate to tell me if I'm >>> wrong. >>> I am developing a code called cafes that studies fluid-particle >>> interactions in a Stokes fluid. For that, we implemented the whole solver >>> in matrix free on a cartesian grid. The preconditioner is naturally a >>> geometrical multigrid for the velocity. The set of matrix free for each >>> level is set by hand (I worked with Matthew on this a long time ago now!). >>> I would have liked to implement telescope on this part. I started it but >>> I didn't get out of it. So I would like to have your help. >>> What I tried to do was to describe my classical multigrid up to a level >>> N1 and then take the coarse solver and apply the telescope preconditioner >>> to it up to a level N2. I don't know if this is the best way but it seemed >>> the most intuitive. The problem is that I could not
Re: [petsc-dev] Telescope usage
Hi Dave, could you explain what you mean by state ? Thanks, Loic Le 17/05/2022 à 11:50, Dave May a écrit : Hi Loic, Can you confirm if your problem has stored state which needs to be repartitioned? Thanks, Dave On Tue 17. May 2022 at 09:07, Loic Gouarin wrote: Le 17/05/2022 à 01:43, Dave May a écrit : On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: On Mon, May 16, 2022 at 9:59 AM Loic Gouarin wrote: Le 16/05/2022 à 21:54, Barry Smith a écrit : On May 16, 2022, at 3:50 PM, Loic Gouarin wrote: Thanks Barry for your quick reply. Le 16/05/2022 à 21:41, Barry Smith a écrit : Loic, From your code it looks like you are using a DM. Is it a DMDA or a shell DM? If it is a DMDA then the process is intended to be pretty straightforward. PCTELESCOPE should create a new DMDA that has the same properties as the coarse grid DMDA but lives on a smaller number of ranks. From this you can just provide multiple levels of coarsening of the DMDA to produce the smaller multigrid problems. It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the DM which is the same of the coarse grid and build all the levels as previously for the levels of PCTELESCOPE ? Yes, I think you should be able to pull out of the coarse level PC (on the subset of ranks) a DM that and then setup coarse levels from that. But you need to look at an example that that uses telescope and probably even the telescope code itself to see all the details. I don't know them anymore. I looked at the source code but It didn't help. I will try to explain more clearly my experiments to understand what I make wrong and you can help more easily. Hi Loic! Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it creates a new DMDA and copies over all the information it can see. However, I think you may have state that is attached to the old DMDA that is not getting redistributed. If this is true, I can help you write a hook that redistributes that state when TELESCOPE maps between distributions. If you have state you need to propagate between different communicators, you will have to use the telescope type which is internally is referred to as TELESCOPE_COARSEDM To activate this type you need the option -pc_telescope_use_coarse_dm Or to call PCTelescopeSetUseCoarseDM() The assumptions of usng this options are described here https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM and source associated with the COARSEDM https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html Using TELESCOPE_COARSEDM you will have access to the hooks Matt is referring to. Thanks Matthew and Dave. I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero instances of a coarse DM were found". And indeed, my coarse solver only has the DM on the coarse mesh and not on the fine level with a coarse DM atttached. I think it's because I use DMKSPSetComputeOperators to create the KSP on each level. So for now, I just added KSP coarse; ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); PC pc_coarse; ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); ierr=PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); but it's not enough. Cheers, Loic Cheers, Dave Thanks, Matt Thanks, Loic Barry Can you let us know more details of what exactly goes wrong with attempts? It is likely straightforward to get things working, perhaps due to our unclear documentation. I essentially have wrong state or NULL matrices into the PCTELESCOPE. Loic Barry On May 16, 2022, at 3:28 PM, Loic Gouarin wrote: Hello, I could have posted my message on the user list but it seems to me that it's more in the petsc development part. Don't hesitate to tell me if I'm wrong. I am developing a code called cafes that studies fluid-particle interactions in a Stokes fluid. For that, we implemented the whole solver in matrix free on a cartesian grid. The preconditioner is naturally a geometrical multigrid for the velocity. The set of matrix free for each level is set by hand (I worked
Re: [petsc-dev] Telescope usage
On Tue 17. May 2022 at 11:56, Loic Gouarin wrote: > Hi Dave, > > could you explain what you mean by state ? > Ah - by state Matt and I mean any auxiliary data stored in the user context passed to KSPSetComputeOperators which is required to define your operator and is distributed. For example, a Vec which is used store coefficients of your PDE. Thanks, > > Loic > Le 17/05/2022 à 11:50, Dave May a écrit : > > Hi Loic, > > Can you confirm if your problem has stored state which needs to be > repartitioned? > > Thanks, > Dave > > On Tue 17. May 2022 at 09:07, Loic Gouarin > wrote: > >> >> Le 17/05/2022 à 01:43, Dave May a écrit : >> >> >> >> On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: >> >>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < >>> loic.goua...@polytechnique.edu> wrote: >>> Le 16/05/2022 à 21:54, Barry Smith a écrit : On May 16, 2022, at 3:50 PM, Loic Gouarin < loic.goua...@polytechnique.edu> wrote: Thanks Barry for your quick reply. Le 16/05/2022 à 21:41, Barry Smith a écrit : Loic, From your code it looks like you are using a DM. Is it a DMDA or a shell DM? If it is a DMDA then the process is intended to be pretty straightforward. PCTELESCOPE should create a new DMDA that has the same properties as the coarse grid DMDA but lives on a smaller number of ranks. From this you can just provide multiple levels of coarsening of the DMDA to produce the smaller multigrid problems. It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the DM which is the same of the coarse grid and build all the levels as previously for the levels of PCTELESCOPE ? Yes, I think you should be able to pull out of the coarse level PC (on the subset of ranks) a DM that and then setup coarse levels from that. But you need to look at an example that that uses telescope and probably even the telescope code itself to see all the details. I don't know them anymore. I looked at the source code but It didn't help. I will try to explain more clearly my experiments to understand what I make wrong and you can help more easily. >>> Hi Loic! >>> >>> Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it >>> creates a new DMDA and copies over all the information it can see. However, >>> I think you may have state that is attached to the old DMDA that is not >>> getting redistributed. If this is true, I can help you write a hook that >>> redistributes that state when TELESCOPE maps between distributions. >>> >> >> If you have state you need to propagate between different communicators, >> you will have to use the telescope type which is internally is referred to >> as >> >> TELESCOPE_COARSEDM >> >> >> To activate this type you need the option >> >> -pc_telescope_use_coarse_dm >> >> Or to call PCTelescopeSetUseCoarseDM() >> The assumptions of usng this options are described here >> https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM >> >> and source associated with the COARSEDM >> >> https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html >> >> Using TELESCOPE_COARSEDM you will have access to the hooks Matt is >> referring to. >> >> Thanks Matthew and Dave. >> >> I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero >> instances of a coarse DM were found". And indeed, my coarse solver only has >> the DM on the coarse mesh and not on the fine level with a coarse DM >> atttached. I think it's because I use DMKSPSetComputeOperators to create >> the KSP on each level. >> >> So for now, I just added >> KSP coarse; >> ierr = PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); >> PC pc_coarse; >> ierr = KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); >> ierr = PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); >> ierr = PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); >> >> but it's not enough. >> >> Cheers, >> >> Loic >> >> >> Cheers, >> Dave >> >> >> >>>Thanks, >>> >>> Matt >>> >>> Thanks, Loic Barry Can you let us know more details of what exactly goes wrong with attempts? It is likely straightforward to get things working, perhaps due to our unclear documentation. I essentially have wrong state or NULL matrices into the PCTELESCOPE. Loic Barry On May 16, 2022, at 3:28 PM, Loic Gouarin < loic.goua...@polytechnique.edu> wrote: Hello, I could have posted my message on the user list but it seems to me that it's more in the petsc development part. Don't hesitate to tell me if I'm wrong. I am developing a code called cafes that studies fluid-particle interactions in a Stokes fluid. For that, we implemented the whole solver in matrix free on a cartesian grid. The preconditioner is naturally a
Re: [petsc-dev] odd log behavior
Can I get some advice on how to add/hack an event now into the reported events? I am noticing that all of the difference in my clean (warmed up) stage comes for KSPSolve: KSPSolve 144 1.0 *2.1745e+00* 1.0 7.50e+10 1.0 0.0e+00 0.0e+00 0.0e+00 42 89 0 0 0 68 98 0 0 0 34511 52718 0 0.00e+000 0.00e+00 100 KSPSolve 144 1.0 *1.2275e+00* 1.0 7.50e+10 1.0 0.0e+00 0.0e+00 0.0e+00 29 89 0 0 0 56 98 0 0 0 61138 * -nan* 0 0.00e+000 0.00e+00 100 Maybe it is easier for me to just run it twice and get my Landau numbers from the full report and get KSPSolve numbers from the nan report. (Not great for a reproducibility doc) Thanks, Mark On Mon, May 16, 2022 at 10:38 PM Mark Adams wrote: > Ok thanks, > > And looking at the actual tests that I am doing more carefully I am not > seeing much difference in the numbers that I care about with or without the > GPU timers. > > And FWIW, from a user experience point of view, having -log_view present > you with a sea of nans by default is a bit disheartening for a first time > user. And they have to send an email and we have to respond > > > On Mon, May 16, 2022 at 9:36 PM Barry Smith wrote: > >> >> Mark, >> >> Yes, Jed has already asked for ways to select certain operations at >> runtime that would get timed via the GPU timer. I am not sure of the ideal >> way to handle it. One possibility is PetscLogGpuTimeEvent(event number) >> which indicates events you want the time logged for. Another is >> PetscLogGpuTimeObjectID(object ID) which indicates all events for a >> particular object class would be logged. In either or both cases the >> PetscLogGPUTime() calls would need to become smarter to determine when they >> are turned on and off depending on the current state of the logging. Since >> the events are nested and there can be multiple objects related to the >> timings I don't have a clear plan in my mind. >> >>Barry >> >> >> On May 16, 2022, at 7:31 PM, Mark Adams wrote: >> >> I am not sure I understand the logic, we print the ratio of max/min. >> I report max and look at the ratio to see if I might be catching some >> load imbalance or whatever. Is there a problem with that workflow? >> >> I assume there is or you would not have done this, so can I add a method >> that I think also has legitimate values? >> >> I did observe a 15% hit on my whole stage with the GPU timers but I would >> still like to take my chances with: >> >> Landau Operator 29 1.0 1.1200e-02 1.0 4.54e+06 1.0 0.0e+00 0.0e+00 >> 0.0e+00 3 0 0 0 0 40 4 0 0 0 405 509 0 0.00e+000 >> 0.00e+00 100 >> Landau Jacobian 15 1.0 7.8189e-03 1.0 4.54e+06 1.0 0.0e+00 0.0e+00 >> 0.0e+00 2 0 0 0 0 28 4 0 0 0 580 731 0 0.00e+000 >> 0.00e+00 100 >> Landau Mass 14 1.0 3.3759e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 1 0 0 0 0 12 0 0 0 0 0 0 0 0.00e+000 >> 0.00e+00 0 >> >> These methods require barriers at the end before they >> call MatSetValuesCOO, which is like < 1% of these total matrix times, so I >> trust the data w/o GPU times set. >> >> (The rest of the stage is the rest of the time step, which is mostly the >> linear solver, and I guess that is where the 15% comes from but I have not >> dug into it) >> >> Thanks, >> Mark >> >> >> On Wed, Apr 27, 2022 at 11:06 AM Barry Smith wrote: >> >>> >>> Only KSPSolve, SNESSolve, TSSolve will have legitimate values. >>> "High-level" functions like PtAP can be asynchronous (meaning the GPU >>> returns to the CPU to do more stuff) before they are complete, hence their >>> timings would be incorrect and so they must be labeled with a special >>> marker and not have numbers for time. >>> >>> Jed reports a possible 10% smaller time for KSPSolve() in this mode in >>> some of his runs, compared to timing correctly the inner operations, and >>> thus feels it is the best default. >>> >>> Barry >>> >>> >>> On Apr 27, 2022, at 10:08 AM, Mark Adams wrote: >>> >>> >>> >>> On Tue, Apr 26, 2022 at 8:00 PM Barry Smith wrote: >>> The current nan output has to be replaced to get the column alignment correct, I just didn't feel like making that change also in the same MR. Something like Unknown or anything that fits in the column space would be fine. It just means for the given run the timing numbers are not meaningful/correct for those events. >>> >>> Just a note, just about every event is NAN for me. My GAMG setup that is >>> all CPU is NAN. High level functions like PtAP as well. >>> That said, adding -log_view_gpu_time is fine. Not worth the churn. >>> >>> This is to obtain the best meaningful results for the outer events per Jed since timing the inner events accurately introduces extra time in the outer events. That is it is not possible to have the best accurate times for both inner events and outer events in the same run. So if you want
Re: [petsc-dev] Telescope usage
Le 17/05/2022 à 12:03, Dave May a écrit : On Tue 17. May 2022 at 11:56, Loic Gouarin wrote: Hi Dave, could you explain what you mean by state ? Ah - by state Matt and I mean any auxiliary data stored in the user context passed to KSPSetComputeOperators which is required to define your operator and is distributed. For example, a Vec which is used store coefficients of your PDE. Yes, I have a context which gives the operators (mult and diagonal) and some other informations to build the matrix free on each multigrid level. You can find the idea here: https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91 It's the definition of the operator used inside DMKSPComputeOperators. Loic Thanks, Loic Le 17/05/2022 à 11:50, Dave May a écrit : Hi Loic, Can you confirm if your problem has stored state which needs to be repartitioned? Thanks, Dave On Tue 17. May 2022 at 09:07, Loic Gouarin wrote: Le 17/05/2022 à 01:43, Dave May a écrit : On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: On Mon, May 16, 2022 at 9:59 AM Loic Gouarin wrote: Le 16/05/2022 à 21:54, Barry Smith a écrit : On May 16, 2022, at 3:50 PM, Loic Gouarin wrote: Thanks Barry for your quick reply. Le 16/05/2022 à 21:41, Barry Smith a écrit : Loic, From your code it looks like you are using a DM. Is it a DMDA or a shell DM? If it is a DMDA then the process is intended to be pretty straightforward. PCTELESCOPE should create a new DMDA that has the same properties as the coarse grid DMDA but lives on a smaller number of ranks. From this you can just provide multiple levels of coarsening of the DMDA to produce the smaller multigrid problems. It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get the DM which is the same of the coarse grid and build all the levels as previously for the levels of PCTELESCOPE ? Yes, I think you should be able to pull out of the coarse level PC (on the subset of ranks) a DM that and then setup coarse levels from that. But you need to look at an example that that uses telescope and probably even the telescope code itself to see all the details. I don't know them anymore. I looked at the source code but It didn't help. I will try to explain more clearly my experiments to understand what I make wrong and you can help more easily. Hi Loic! Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it creates a new DMDA and copies over all the information it can see. However, I think you may have state that is attached to the old DMDA that is not getting redistributed. If this is true, I can help you write a hook that redistributes that state when TELESCOPE maps between distributions. If you have state you need to propagate between different communicators, you will have to use the telescope type which is internally is referred to as TELESCOPE_COARSEDM To activate this type you need the option -pc_telescope_use_coarse_dm Or to call PCTelescopeSetUseCoarseDM() The assumptions of usng this options are described here https://petsc.org/main/docs/manualpages/PC/PCTelescopeSetUseCoarseDM and source associated with the COARSEDM https://petsc.org/main/src/ksp/pc/impls/telescope/telescope_coarsedm.c.html Using TELESCOPE_COARSEDM you will have access to the hooks Matt is referring to. Thanks Matthew and Dave. I used TELESCOPE_COARSEDM in my previous tests but it tells me that "Zero instances of a coarse DM were found". And indeed, my coarse solver only has the DM on the coarse mesh and not on the fine level with a coarse DM atttached. I think it's because I use DMKSPSetComputeOperators to create the KSP on each level. So for now, I just added KSP coarse; ierr=PCMGGetCoarseSolve(pc_i, &coarse);CHKERRQ(ierr); PC pc_coarse; ierr=KSPGetPC(coarse, &pc_coarse);CHKERRQ(ierr); ierr=PCSetType(pc_coarse, PCTELESCOPE);CHKERRQ(ierr); ierr=PCTelescopeSetUseCoarseDM(pc_coarse, PETSC_TRUE);CHKERRQ(ierr); but it's not enough. Cheers, Loic Cheers, Dave Thanks, Matt
Re: [petsc-dev] odd log behavior
Oh, this is not hard:
/* /\* Put NaN into the time for all events that may not be time
accurately since they may happen asynchronously on the GPU *\/ */
/* #if defined(PETSC_HAVE_DEVICE) */
/* if (!PetscLogGpuTimeFlag && petsc_gflops > 0) { */
/* memcpy(&gmaxt,&nas,sizeof(PetscLogDouble)); */
/* PetscCall(PetscEventRegLogGetEvent(stageLog->eventLog, name,
&eventid)); */
/* if (eventid != SNES_Solve && eventid != KSP_Solve && eventid !=
TS_Step && eventid != TAO_Solve) { */
/* memcpy(&mint,&nas,sizeof(PetscLogDouble)); */
/* memcpy(&maxt,&nas,sizeof(PetscLogDouble)); */
/* } */
/* } */
/* #endif */
On Tue, May 17, 2022 at 7:17 AM Mark Adams wrote:
> Can I get some advice on how to add/hack an event now into the reported
> events?
>
> I am noticing that all of the difference in my clean (warmed up) stage
> comes for KSPSolve:
>
> KSPSolve 144 1.0 *2.1745e+00* 1.0 7.50e+10 1.0 0.0e+00
> 0.0e+00 0.0e+00 42 89 0 0 0 68 98 0 0 0 34511 52718 0
> 0.00e+000 0.00e+00 100
> KSPSolve 144 1.0 *1.2275e+00* 1.0 7.50e+10 1.0 0.0e+00
> 0.0e+00 0.0e+00 29 89 0 0 0 56 98 0 0 0 61138 * -nan* 0
> 0.00e+000 0.00e+00 100
>
> Maybe it is easier for me to just run it twice and get my Landau numbers
> from the full report and get KSPSolve numbers from the nan report.
> (Not great for a reproducibility doc)
>
> Thanks,
> Mark
>
>
>
> On Mon, May 16, 2022 at 10:38 PM Mark Adams wrote:
>
>> Ok thanks,
>>
>> And looking at the actual tests that I am doing more carefully I am not
>> seeing much difference in the numbers that I care about with or without the
>> GPU timers.
>>
>> And FWIW, from a user experience point of view, having -log_view present
>> you with a sea of nans by default is a bit disheartening for a first time
>> user. And they have to send an email and we have to respond
>>
>>
>> On Mon, May 16, 2022 at 9:36 PM Barry Smith wrote:
>>
>>>
>>> Mark,
>>>
>>> Yes, Jed has already asked for ways to select certain operations at
>>> runtime that would get timed via the GPU timer. I am not sure of the ideal
>>> way to handle it. One possibility is PetscLogGpuTimeEvent(event number)
>>> which indicates events you want the time logged for. Another is
>>> PetscLogGpuTimeObjectID(object ID) which indicates all events for a
>>> particular object class would be logged. In either or both cases the
>>> PetscLogGPUTime() calls would need to become smarter to determine when they
>>> are turned on and off depending on the current state of the logging. Since
>>> the events are nested and there can be multiple objects related to the
>>> timings I don't have a clear plan in my mind.
>>>
>>>Barry
>>>
>>>
>>> On May 16, 2022, at 7:31 PM, Mark Adams wrote:
>>>
>>> I am not sure I understand the logic, we print the ratio of max/min.
>>> I report max and look at the ratio to see if I might be catching some
>>> load imbalance or whatever. Is there a problem with that workflow?
>>>
>>> I assume there is or you would not have done this, so can I add a method
>>> that I think also has legitimate values?
>>>
>>> I did observe a 15% hit on my whole stage with the GPU timers but I
>>> would still like to take my chances with:
>>>
>>> Landau Operator 29 1.0 1.1200e-02 1.0 4.54e+06 1.0 0.0e+00 0.0e+00
>>> 0.0e+00 3 0 0 0 0 40 4 0 0 0 405 509 0 0.00e+000
>>> 0.00e+00 100
>>> Landau Jacobian 15 1.0 7.8189e-03 1.0 4.54e+06 1.0 0.0e+00 0.0e+00
>>> 0.0e+00 2 0 0 0 0 28 4 0 0 0 580 731 0 0.00e+000
>>> 0.00e+00 100
>>> Landau Mass 14 1.0 3.3759e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>> 0.0e+00 1 0 0 0 0 12 0 0 0 0 0 0 0 0.00e+000
>>> 0.00e+00 0
>>>
>>> These methods require barriers at the end before they
>>> call MatSetValuesCOO, which is like < 1% of these total matrix times, so I
>>> trust the data w/o GPU times set.
>>>
>>> (The rest of the stage is the rest of the time step, which is mostly the
>>> linear solver, and I guess that is where the 15% comes from but I have not
>>> dug into it)
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>> On Wed, Apr 27, 2022 at 11:06 AM Barry Smith wrote:
>>>
Only KSPSolve, SNESSolve, TSSolve will have legitimate values.
"High-level" functions like PtAP can be asynchronous (meaning the GPU
returns to the CPU to do more stuff) before they are complete, hence their
timings would be incorrect and so they must be labeled with a special
marker and not have numbers for time.
Jed reports a possible 10% smaller time for KSPSolve() in this mode
in some of his runs, compared to timing correctly the inner operations, and
thus feels it is the best default.
Barry
On Apr 27, 2022, at 10:08 AM, Mark Adams wrote:
On Tue, Apr 26, 2022 at 8:00 PM Barry Smith wrote:
>
> The curre
Re: [petsc-dev] odd log behavior
Python users including myself would love NaN since NaN is the default missing value marker for reasons of computational speed and convenience. For example, if you load these values into pandas, no extra code is needed to handle them. Other choices such as N/A would require some extra work for text replacement. The -nan looks a bit weird. There should be a way to get rid of the - sign. Hong (Mr.) > On Apr 26, 2022, at 10:52 AM, Jacob Faibussowitsch > wrote: > > There is an automatic warning that shows when you do run with > `-log_view_gpu_time`, but perhaps there should also be an automatic warning > when *not* running with it. It is unfortunate that NaN is the value printed > as this implies a bug but AFAIK it is unavoidable (Barry can say more on this > though). > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > >> On Apr 26, 2022, at 09:48, Jose E. Roman wrote: >> >> You have to add -log_view_gpu_time >> See https://gitlab.com/petsc/petsc/-/merge_requests/5056 >> >> Jose >> >> >>> El 26 abr 2022, a las 16:39, Mark Adams escribió: >>> >>> I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing >>> data except the two 'Solve' lines. >>> Just cg/jacobi on snes/ex56. >>> >>> Any ideas? >>> >>> VecTDot2 1.0 nan nan 1.20e+01 1.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 100 >>> VecNorm2 1.0 nan nan 1.00e+01 1.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 100 >>> VecCopy2 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 0 >>> VecSet 5 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 0 >>> VecAXPY4 1.0 nan nan 2.40e+01 1.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 1 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 100 >>> VecPointwiseMult 1 1.0 nan nan 3.00e+00 1.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 100 >>> KSPSetUp 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 >>> 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000 0.00e+00 >>> 0 >>> KSPSolve 1 1.0 4.0514e-04 1.0 5.50e+01 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 0 0 0 0 2 0 0 0 0 0-nan 0 0.00e+000 >>> 0.00e+00 100 >>> SNESSolve 1 1.0 2.2128e-02 1.0 5.55e+05 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 72 56 0 0 0 100100 0 0 025-nan 0 0.00e+000 >>> 0.00e+00 0 >> >
Re: [petsc-dev] Telescope usage
Dear Loic, I can confirm that PCTELESCOPE works nicely when using KSPSetComputeOperators. Here is an example petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2 There is no special code in ex29.c to make the above options run. I must also apologize for sending you down the garden path with COARSEDM. That really isn't what you want to use - I am sorry for the confusion on my part. COARSEDM is intended for the case when YOU (or rather your coarse DM) define the sub-communicator, rather than having PCTELESCOPE define the sub-communicator. I believe that PCTELESCOPE should works with DMKSPSetComputeOperators(). The function you provide to DMKSPSetComputeOperators() is unusual in my opinion. Normally the method provided to DMKSPSetComputeOperators() just "assembles" (assembles meaning insert non-zero values for an AIJ or just calls MatAssemblyBegin/End for a matrix-free operator). However your method seems to set sizes, set methods and call SetFromOptions. These are typically things which should done when the matrix is created - not when it is being "assembled". Maybe this is part of the reason your code isn't playing nicely with telescope. I think the code would be cleaner and if you overloaded your DMCreateMatrix() with a method which would return your matrix-free MATSHELL. Yes I realize I am not directly helping solve your problem but maybe indirectly I am. Matt? Cheers, Dave On Tue, 17 May 2022 at 13:18, Loic Gouarin wrote: > > Le 17/05/2022 à 12:03, Dave May a écrit : > > > > On Tue 17. May 2022 at 11:56, Loic Gouarin > wrote: > >> Hi Dave, >> >> could you explain what you mean by state ? >> > > Ah - by state Matt and I mean any auxiliary data stored in the user > context passed to KSPSetComputeOperators which is required to define your > operator and is distributed. For example, a Vec which is used store > coefficients of your PDE. > > Yes, I have a context which gives the operators (mult and diagonal) and > some other informations to build the matrix free on each multigrid level. > > You can find the idea here: > https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91 > > It's the definition of the operator used inside DMKSPComputeOperators. > > Loic > > > > > Thanks, >> >> Loic >> Le 17/05/2022 à 11:50, Dave May a écrit : >> >> Hi Loic, >> >> Can you confirm if your problem has stored state which needs to be >> repartitioned? >> >> Thanks, >> Dave >> >> On Tue 17. May 2022 at 09:07, Loic Gouarin < >> loic.goua...@polytechnique.edu> wrote: >> >>> >>> Le 17/05/2022 à 01:43, Dave May a écrit : >>> >>> >>> >>> On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: >>> On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < loic.goua...@polytechnique.edu> wrote: > Le 16/05/2022 à 21:54, Barry Smith a écrit : > > > > On May 16, 2022, at 3:50 PM, Loic Gouarin < > loic.goua...@polytechnique.edu> wrote: > > Thanks Barry for your quick reply. > Le 16/05/2022 à 21:41, Barry Smith a écrit : > > > Loic, > > From your code it looks like you are using a DM. Is it a DMDA or a > shell DM? If it is a DMDA then the process is intended to be pretty > straightforward. PCTELESCOPE should create a new DMDA that has the same > properties as the coarse grid DMDA but lives on a smaller number of ranks. > From this you can just provide multiple levels of coarsening of the DMDA > to > produce the smaller multigrid problems. > > It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get > the DM which is the same of the coarse grid and build all the levels as > previously for the levels of PCTELESCOPE ? > > > Yes, I think you should be able to pull out of the coarse level PC > (on the subset of ranks) a DM that and then setup coarse levels from that. > But you need to look at an example that that uses telescope and probably > even the telescope code itself to see all the details. I don't know them > anymore. > > I looked at the source code but It didn't help. I will try to explain > more clearly my experiments to understand what I make wrong and you can > help more easily. > Hi Loic! Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, it creates a new DMDA and copies over all the information it can see. However, I think you may have state that is attached to the old DMDA that is not getting redistributed. If this is true, I can help you write a hook that redistributes that state when TELESCOPE maps between distributions. >>> >>> If you have state you need to propagate between different communicators, >>> you will have to use the telescope type which is internally is
Re: [petsc-dev] Telescope usage
On Tue, May 17, 2022 at 4:53 AM Dave May wrote: > Dear Loic, > > I can confirm that PCTELESCOPE works nicely when using > KSPSetComputeOperators. > Here is an example > > petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg > -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope > -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 > -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2 > > There is no special code in ex29.c to make the above options run. > > I must also apologize for sending you down the garden path with COARSEDM. > That really isn't what you want to use - I am sorry for the confusion on my > part. > COARSEDM is intended for the case when YOU (or rather your coarse DM) > define the sub-communicator, rather than having PCTELESCOPE define the > sub-communicator. > > I believe that PCTELESCOPE should works with DMKSPSetComputeOperators(). > > The function you provide to DMKSPSetComputeOperators() is unusual in my > opinion. > Normally the method provided to DMKSPSetComputeOperators() just > "assembles" (assembles meaning insert non-zero values for an AIJ or just > calls MatAssemblyBegin/End for a matrix-free operator). > However your method seems to set sizes, set methods and call > SetFromOptions. These are typically things which should done when the > matrix is created - not when it is being "assembled". > > Maybe this is part of the reason your code isn't playing nicely with > telescope. > I think the code would be cleaner and if you overloaded your > DMCreateMatrix() with a method which would return your matrix-free MATSHELL. > > Yes I realize I am not directly helping solve your problem but maybe > indirectly I am. Matt? > Yes, I agree with Dave. This refactoring, even if it does not solve the problem, should allow us to understand it better. Does the change make sense? Thanks, Matt > Cheers, > Dave > > > On Tue, 17 May 2022 at 13:18, Loic Gouarin > wrote: > >> >> Le 17/05/2022 à 12:03, Dave May a écrit : >> >> >> >> On Tue 17. May 2022 at 11:56, Loic Gouarin < >> loic.goua...@polytechnique.edu> wrote: >> >>> Hi Dave, >>> >>> could you explain what you mean by state ? >>> >> >> Ah - by state Matt and I mean any auxiliary data stored in the user >> context passed to KSPSetComputeOperators which is required to define your >> operator and is distributed. For example, a Vec which is used store >> coefficients of your PDE. >> >> Yes, I have a context which gives the operators (mult and diagonal) and >> some other informations to build the matrix free on each multigrid level. >> >> You can find the idea here: >> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91 >> >> It's the definition of the operator used inside DMKSPComputeOperators. >> >> Loic >> >> >> >> >> Thanks, >>> >>> Loic >>> Le 17/05/2022 à 11:50, Dave May a écrit : >>> >>> Hi Loic, >>> >>> Can you confirm if your problem has stored state which needs to be >>> repartitioned? >>> >>> Thanks, >>> Dave >>> >>> On Tue 17. May 2022 at 09:07, Loic Gouarin < >>> loic.goua...@polytechnique.edu> wrote: >>> Le 17/05/2022 à 01:43, Dave May a écrit : On Tue, 17 May 2022 at 00:12, Matthew Knepley wrote: > On Mon, May 16, 2022 at 9:59 AM Loic Gouarin < > loic.goua...@polytechnique.edu> wrote: > >> Le 16/05/2022 à 21:54, Barry Smith a écrit : >> >> >> >> On May 16, 2022, at 3:50 PM, Loic Gouarin < >> loic.goua...@polytechnique.edu> wrote: >> >> Thanks Barry for your quick reply. >> Le 16/05/2022 à 21:41, Barry Smith a écrit : >> >> >> Loic, >> >> From your code it looks like you are using a DM. Is it a DMDA or >> a shell DM? If it is a DMDA then the process is intended to be pretty >> straightforward. PCTELESCOPE should create a new DMDA that has the same >> properties as the coarse grid DMDA but lives on a smaller number of >> ranks. >> From this you can just provide multiple levels of coarsening of the DMDA >> to >> produce the smaller multigrid problems. >> >> It's a DMDA. So what you mean is to take the KSP of PCTELESCOPE, get >> the DM which is the same of the coarse grid and build all the levels as >> previously for the levels of PCTELESCOPE ? >> >> >> Yes, I think you should be able to pull out of the coarse level PC >> (on the subset of ranks) a DM that and then setup coarse levels from >> that. >> But you need to look at an example that that uses telescope and probably >> even the telescope code itself to see all the details. I don't know them >> anymore. >> >> I looked at the source code but It didn't help. I will try to explain >> more clearly my experiments to understand what I make wrong and you can >> help more easily. >> > Hi Loic! > > Here is my guess. When TELESCOPE redistributes the coarse grid DMDA, > it c
Re: [petsc-dev] Telescope usage
Thanks Dave for your time and this detailed reply. My implementation is old and should probably be updated. I will read your email carefully and I come back to you with a better implementation. Thanks, Loic 17 mai 2022 16:53:37 Dave May : > Dear Loic, > > I can confirm that PCTELESCOPE works nicely when using KSPSetComputeOperators. > Here is an example > > petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg > -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope > -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 > -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2 > > There is no special code in ex29.c to make the above options run. > > I must also apologize for sending you down the garden path with COARSEDM. > That really isn't what you want to use - I am sorry for the confusion on my > part. > COARSEDM is intended for the case when YOU (or rather your coarse DM) define > the sub-communicator, rather than having PCTELESCOPE define the > sub-communicator. > > I believe that PCTELESCOPE should works with DMKSPSetComputeOperators(). > > The function you provide to DMKSPSetComputeOperators() is unusual in my > opinion. > Normally the method provided to DMKSPSetComputeOperators() just "assembles" > (assembles meaning insert non-zero values for an AIJ or just calls > MatAssemblyBegin/End for a matrix-free operator). > However your method seems to set sizes, set methods and call SetFromOptions. > These are typically things which should done when the matrix is created - not > when it is being "assembled". > > Maybe this is part of the reason your code isn't playing nicely with > telescope. > I think the code would be cleaner and if you overloaded your DMCreateMatrix() > with a method which would return your matrix-free MATSHELL. > > Yes I realize I am not directly helping solve your problem but maybe > indirectly I am. Matt? > > Cheers, > Dave > > > On Tue, 17 May 2022 at 13:18, Loic Gouarin > wrote: >> >> >> Le 17/05/2022 à 12:03, Dave May a écrit : >> >> >> On Tue 17. May 2022 at 11:56, Loic Gouarin >> wrote: >>> >>> Hi Dave, >>> >>> could you explain what you mean by state ? >>> >> >> Ah - by state Matt and I mean any auxiliary data stored in the user context >> passed to KSPSetComputeOperators which is required to define your operator >> and is distributed. For example, a Vec which is used store coefficients of >> your PDE. >> >> Yes, I have a context which gives the operators (mult and diagonal) and some >> other informations to build the matrix free on each multigrid level. >> >> You can find the idea here: >> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91 >> >> It's the definition of the operator used inside DMKSPComputeOperators. >> >> Loic >> >> >> >> >>> Thanks, >>> >>> Loic >>> >>> Le 17/05/2022 à 11:50, Dave May a écrit : >>> Hi Loic, >>> >>> Can you confirm if your problem has stored state which needs to be >>> repartitioned? >>> >>> Thanks, >>> Dave >>> >>> On Tue 17. May 2022 at 09:07, Loic Gouarin >>> wrote: …
Re: [petsc-dev] Telescope usage
17 mai 2022 17:53:46 Matthew Knepley : > On Tue, May 17, 2022 at 4:53 AM Dave May wrote: >> Dear Loic, >> >> I can confirm that PCTELESCOPE works nicely when using >> KSPSetComputeOperators. >> Here is an example >> >> petsc-3.17.1/src/ksp/ksp/tutorials $ mpiexec -n 4 ./ex29 -pc_type mg >> -pc_mg_levels 2 -ksp_view -mg_coarse_pc_type telescope >> -mg_coarse_pc_telescope_reduction_factor 4 -da_grid_x 65 -da_grid_y 65 >> -mg_coarse_telescope_pc_type mg -mg_coarse_telescope_pc_mg_levels 2 >> >> There is no special code in ex29.c to make the above options run. >> >> I must also apologize for sending you down the garden path with COARSEDM. >> That really isn't what you want to use - I am sorry for the confusion on my >> part. >> COARSEDM is intended for the case when YOU (or rather your coarse DM) define >> the sub-communicator, rather than having PCTELESCOPE define the >> sub-communicator. >> >> I believe that PCTELESCOPE should works with DMKSPSetComputeOperators(). >> >> The function you provide to DMKSPSetComputeOperators() is unusual in my >> opinion. >> Normally the method provided to DMKSPSetComputeOperators() just "assembles" >> (assembles meaning insert non-zero values for an AIJ or just calls >> MatAssemblyBegin/End for a matrix-free operator). >> However your method seems to set sizes, set methods and call SetFromOptions. >> These are typically things which should done when the matrix is created - >> not when it is being "assembled". >> >> Maybe this is part of the reason your code isn't playing nicely with >> telescope. >> I think the code would be cleaner and if you overloaded your >> DMCreateMatrix() with a method which would return your matrix-free MATSHELL. >> >> Yes I realize I am not directly helping solve your problem but maybe >> indirectly I am. Matt? > > Yes, I agree with Dave. This refactoring, even if it does not solve the > problem, should allow us to understand it better. > Does the change make sense? Yes I think! Thanks, Loïc > > Thanks, > > Matt > >> Cheers, >> Dave >> >> >> On Tue, 17 May 2022 at 13:18, Loic Gouarin >> wrote: >>> >>> >>> Le 17/05/2022 à 12:03, Dave May a écrit : >>> >>> >>> On Tue 17. May 2022 at 11:56, Loic Gouarin >>> wrote: Hi Dave, could you explain what you mean by state ? >>> >>> Ah - by state Matt and I mean any auxiliary data stored in the user context >>> passed to KSPSetComputeOperators which is required to define your operator >>> and is distributed. For example, a Vec which is used store coefficients of >>> your PDE. >>> >>> Yes, I have a context which gives the operators (mult and diagonal) and >>> some other informations to build the matrix free on each multigrid level. >>> >>> You can find the idea here: >>> https://github.com/gouarin/cafes/blob/master/cafes/problem/stokes.hpp#L49-L91 >>> >>> It's the definition of the operator used inside DMKSPComputeOperators. >>> >>> Loic >>> >>> >>> >>> Thanks, Loic Le 17/05/2022 à 11:50, Dave May a écrit : … > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/[http://www.cse.buffalo.edu/~knepley/]
