You may be able to use https://petsc.org/main/manualpages/PC/PCSetApplicationContext/#pcsetapplicationcontext or KSPSetApplicationContext() or PetscObjectCompose() (this may be better since it can be eventually merged into PETSc).
Barry > On Jul 29, 2023, at 7:54 AM, Carl-Johan Thore <carl-johan.th...@liu.se> wrote: > > Hi again, > > I’ve now managed to get PCMG with PCREDISTRIBUTE to work on multiple ranks, > and convergence looks good on my > examples. (the modification to redistribute.c was trivial – just skip the > case “if (size > 1) {“ in PCSetUp_Redistribute). > > The final step is your suggestion > > “2) the only problem with 1 is it is likely to be poorly load balanced (but > you can make some runs to see how imbalanced it is, that will depend exactly > on what parts are locked and what MPI processes they are on). So if it is > poorly balanced then you would need to get out of redistribute.c a mapping > for each kept dof to what MPI rank it is moved to and use that to move the > entries in the reduced interpolation you have created. “ > > My idea for this is to do “VecScatterBegin(red->scatter …” on my vector > indicating fixed or free DOFs to redistribute it in the same way as the > global matrix is redistributed. > It is tempting to do something like “KSPGetPC(ksp, &pc); PC_Redistribute > *red = (PC_Redistribute *)pc->data;” in my main code, and then do the scatter > with the red object. But this is not directly > possible, resulting in PC_Redistribute not defined and so on. Another > alternative is perhaps to pass my vector into the PC via the “optional > user-defined context” in the _p_PC context. > Do you have any suggestions or hints for a good (efficient and non-intrusive) > way to “get out of redistribute.c a mapping …”? > > /Carl-Johan > > > From: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> > Sent: Tuesday, July 25, 2023 4:49 PM > To: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > Ok thanks! Good to know I'm not doing anything wrong on the PETSc-side, so to > speak. It seems like it could be related to WSL then > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: 25 July 2023 16:39:37 > To: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > No idea why it would be particularly slow. Probably some issue with making > the PETSc shared library, normally that would take a few seconds on Linux or > a Mac. > > > > > On Jul 25, 2023, at 6:12 AM, Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> wrote: > > Hi again, > > I’ve now gotten PCMG with PCREDISTRIBUTE to work nicely on simple examples on > one core > (attached ppt). I therefore plan to test with multiple cores by first > modifying redistribute.c such > that the dofs are not moved around as you suggested. For this I wonder what > is the recommended way > of modifying a single file in my PETSc-installation? I’ve tried running make > in the folder containing redistribute.c: > > <image001.png> > > This works but is terribly slow on my computer (google indicates that this > could be related > to me using Windows Subsystem for Linux, but I didn’t succeed with the > suggestions) > > Kind regards, > Carl-Johan > > > > From: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> > Sent: Sunday, July 2, 2023 8:07 PM > To: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > "NN" is used to construct the row indices passed to matzerorowscolumns. That > works fine, so at least in that sense NN is correct. But I should probably > compare my IS:es with those row indices as well. > > Yes, making a tiny problem is a good idea. > > Thanks for taking time to look at this > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: 02 July 2023 18:58:50 > To: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> > Cc: petsc-users <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > Are you sure the NN is correct? I cannot see how you set that and know > that it exactly matches the way PCREDISTRIBUTE selects rows? > > I suggest making a tiny problem with artificial matrix values that you > select to "slice" of parts of the grid, so you can see exactly on the grid > that the selected rows and columns are correct as you expect. > > > On Jul 2, 2023, at 2:16 AM, Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> wrote: > > Hi, > > I tried your suggestion > > “1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof > between MPI ranks, just have it remove the locked rows/columns (to start just > run on one MPI rank since then nothing is moved) Then in your code you just > need to pull out the appropriate rows and columns of the interpolation that > correspond to the dof you have kept and pass this smaller interpolation to > the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The > MG convergence should be just as good as on the full system.” > > from below and got the size of the interpolation matrix correct. But the > convergence doesn’t seem right. In the attached .txt-file the code without > redistribute convergences in 8 > FGMRES iterations whereas with redistribute it takes 25 (I’ve tested this on > various meshes and the redistribute-code consistently performs much worse in > terms of number of iterations). The code without redistribute is very well > tested and always performs very well, so I’m fairly certain the error is in > my new code. > > Would you be willing to take a quick look at the attached code snippet to see > if I’m doing some obvious mistake? > > Kind regards, > Carl-Johan > > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: Friday, June 30, 2023 5:21 PM > To: Matthew Knepley <knep...@gmail.com <mailto:knep...@gmail.com>> > Cc: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>>; petsc-users@mcs.anl.gov > <mailto:petsc-users@mcs.anl.gov> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > > > On Jun 30, 2023, at 10:22 AM, Matthew Knepley <knep...@gmail.com > <mailto:knep...@gmail.com>> wrote: > > On Fri, Jun 30, 2023 at 10:16 AM Carl-Johan Thore via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > Thanks for the quick reply and the suggestions! > > “ … you should first check that the PCMG works quite well “ > > Yes, the PCMG works very well for the full system. > > “I am guessing that your code is slightly different than ex42.c because you > take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1.” > > Yes, it’s slightly different so problem 2 should be solved. > > It looked somewhat complicated to get PCMG to work with redistribute, so I’ll > try with PCGAMG first > (it ran immediately with redistribute, but was slower than PCMG on my, very > small, test problem. I’ll try > to tune the settings). > > A related question: I’m here using a DMDA for a structured grid but I’m > locking so many DOFs that for many of the elements > all DOFs are locked. In such a case could it make sense to switch/convert the > DMDA to a DMPlex containing only those > elements that actually have DOFs? > > Possibly, but if you are doing FD, then there is built-in topology in DMDA > that is not present in Plex, so > finding the neighbors in the right order is harder (possible, but harder, we > address this in some new work that is not yet merged). There is also > structured adaptive support with DMForest, but this also does not preserve > the stencil. > > The efficiency of active set VI solvers in PETSc demonstrates to me that > solving reduced systems can be done efficiently with geometric multigrid > using a structured grid so I would not suggest giving up on what you started. > > You can do it in two steps > > 1) Use PCREDISTRIBUTE but hack the code in redistribute.c to not move dof > between MPI ranks, just have it remove the locked rows/columns (to start just > run on one MPI rank since then nothing is moved) Then in your code you just > need to pull out the appropriate rows and columns of the interpolation that > correspond to the dof you have kept and pass this smaller interpolation to > the inner KSP PCMG. This is straightforward and like what is in DMSetVI. The > MG convergence should be just as good as on the full system. > > 2) the only problem with 1 is it is likely to be poorly load balanced (but > you can make some runs to see how imbalanced it is, that will depend exactly > on what parts are locked and what MPI processes they are on). So if it is > poorly balanced then you would need to get out of redistribute.c a mapping > for each kept dof to what MPI rank it is moved to and use that to move the > entries in the reduced interpolation you have created. > > If you do succeed it would actually be useful code that we could add to > PCREDISTRIBUTE for more general use by others. > > Barry > > > > > > Thanks, > > Matt > > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: Friday, June 30, 2023 3:57 PM > To: Carl-Johan Thore <carl-johan.th...@liu.se > <mailto:carl-johan.th...@liu.se>> > Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > Subject: Re: [petsc-users] PCMG with PCREDISTRIBUTE > > > Oh, I forgot to mention you should first check that the PCMG works quite > well for the full system (without the PCREDISTRIBUTE); the convergence > on the redistributed system (assuming you did all the work to get PCMG to > work for you) should be very similar to (but not measurably better) than the > convergence on the full system. > > > > > On Jun 30, 2023, at 9:17 AM, Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > > ex42.c provides directly the interpolation/restriction needed to move > between levels in the loop > > for (k = 1; k < nlevels; k++) { > PetscCall(DMCreateInterpolation(da_list[k - 1], da_list[k], &R, NULL)); > PetscCall(PCMGSetInterpolation(pc, k, R)); > PetscCall(MatDestroy(&R)); > } > > The more standard alternative to this is to call KSPSetDM() and have the PCMG > setup use the DM > to construct the interpolations (I don't know why ex42.c does this > construction itself instead of having the KSPSetDM() process handle it but > that doesn't matter). The end result is the same in both cases. > > Since PCREDISTRIBUTE builds its own new matrix (by using only certain rows > and columns of the original matrix) the original interpolation > cannot be used for two reasons > > 1) (since it is for the full system) It is for the wrong problem. > > 2) In addition, if you ran with ex42.c the inner KSP does not have access to > the interpolation that was constructed so you could not get PCMG to to work > as indicated below. > > I am guessing that your code is slightly different than ex42.c because you > take the interpolation matrix provided by the DM > and give it to the inner KSP PCMG?. So you solve problem 2 but not problem 1. > > So the short answer is that there is no "canned" way to use the PCMG process > trivially with PCDISTRIBUTE. > > To do what you want requires two additional steps > > 1) after you construct the full interpolation matrix (by using the DM) you > need to remove the rows associated with the dof that have been removed by the > "locked" variables (and the columns that are associated with coarse grid > points that live on the removed points) so that the interpolation is the > correct "size" for the smaller problem > > 2) since PCREDISTRIBUTE actually moves dof of freedom between MPI processes > for load balancing after it has removed the locked variables you would need > to do the exact same movement for the rows of the interpolation matrix that > you have constructed (after you have removed the "locked" rows of the > interpolation. > > Lots of bookkeeping to acheive 1 and 2 but conceptually simple. > > As an experiment you can try using PCGAMG on the redistributed matrix > -redistribute_pc_type gamg to use algebraic multigrid just to see the time > and convergence rates. Since GAMG creates its own interpolation based on the > matrix and it will be built on the smaller redistributed matrix there will no > issue with the wrong "sized" interpolation. Of course you have the overhead > of algebraic multigrid and cannot take advantage of geometric multigrid. The > GAMG approach may be satisfactory to your needs. > > If you are game for looking more closely at using redistribute with geometric > multigrid and PETSc (which will require digging into PETSc source code and > using internal information in the PETSc source code) you can start by looking > at how we solve variational problems with SNES using reduced space active set > methods. SNESVINEWTONRSLS /src/snes/impls/vi/rs/virs.c This code solves > problem 1 see() it builds the entire interpolation and then pulls out the > required non-locked part. Reduced space active set methods essentially lock > the constrained dof and solve a smaller system without those dof at each > iteration. > > But it does not solve problem 2. Moving the rows of the "smaller" > interpolation to the correct MPI process based on where PCREDISTRIBUTE moved > rows. To do this would requring looking at the PCREDISTRUBUTE code and > extracting the information of where each row was moving and performing the > process for the interpolation matrix. > src/ksp/pc/impls/redistribute/redistribute.c > > Barry > > > > > > > > > > > On Jun 30, 2023, at 8:21 AM, Carl-Johan Thore via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > > Hi, > > I'm trying to run an iterative solver (FGMRES for example) with PCMG as > preconditioner. The setup of PCMG > is done roughly as in ex42 of the PETSc-tutorials > (https://petsc.org/main/src/ksp/ksp/tutorials/ex42.c.html). > Since I have many locked degrees-of-freedom I would like to use > PCREDISTRIBUTE. However, this > results in (30039 is the number of DOFs after redistribute and 55539 the > number before): > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Matrix dimensions of A and P are incompatible for > MatProductType PtAP: A 30039x30039, P 55539x7803 > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.19.0-238-g512d1ae6db4 GIT > Date: 2023-04-24 16:37:00 +0200 > [0]PETSC ERROR: topopt on a arch-linux-c-opt Fri Jun 30 13:28:41 2023 > [0]PETSC ERROR: Configure options COPTFLAGS="-O3 -march=native" > CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" > CUDAOPTFLAGS=-O3 --with-cuda --with-cusp --with-debugging=0 > --download-scalapack --download-hdf5 --download-zlib --download-mumps > --download-parmetis --download-metis --download-ptscotch --download-hypre > --download-spai > [0]PETSC ERROR: #1 MatProductSetFromOptions_Private() at > /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:420 > [0]PETSC ERROR: #2 MatProductSetFromOptions() at > /mnt/c/mathware/petsc/src/mat/interface/matproduct.c:541 > [0]PETSC ERROR: #3 MatPtAP() at > /mnt/c/mathware/petsc/src/mat/interface/matrix.c:9868 > [0]PETSC ERROR: #4 MatGalerkin() at > /mnt/c/mathware/petsc/src/mat/interface/matrix.c:10899 > [0]PETSC ERROR: #5 PCSetUp_MG() at > /mnt/c/mathware/petsc/src/ksp/pc/impls/mg/mg.c:1029 > [0]PETSC ERROR: #6 PCSetUp() at > /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #7 KSPSetUp() at > /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #8 PCSetUp_Redistribute() at > /mnt/c/mathware/petsc/src/ksp/pc/impls/redistribute/redistribute.c:327 > [0]PETSC ERROR: #9 PCSetUp() at > /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994 > [0]PETSC ERROR: #10 KSPSetUp() at > /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406 > [0]PETSC ERROR: #11 KSPSolve_Private() at > /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:824 > [0]PETSC ERROR: #12 KSPSolve() at > /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:1070 > > It’s clear what happens I think, and it kind of make since not all levels are > redistributed as they should (?). > Is it possible to use PCMG with PCREDISTRIBUTE in an easy way? > > Kind regards, > Carl-Johan > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> > > > <pcmg_redistribute_experiment_cantilever.pptx>