Ok, thanks. Definitely more than I expected. It is easy to add the support you requested. I'll push a branch latter today.
Barry > On Aug 21, 2023, at 3:28 PM, Jonas Lundgren <jonas.lundg...@liu.se> wrote: > > Dear Barry, > > I have tried what you suggested on a (not too large) test example on a bigger > cluster that I have access to, using -redistribute_ksp_initial_guess_nonzero > 0 and 1, respectively. The average timings during the 5 first (design) > iterations are 8.7 s (state) and 7.9 s (adjoint) for the case with zero > initial guesses and 5.0 s (state) and 5.7 s (adjoint) for the cases with > nonzero initial guesses. These solvings are the bottleneck of my program, > accounting for about 60-90% of the total computational time, depending on > various parameters. The program is basically consisting of the loop: solve > state > solve adjoint > update design > repeat. This is repeated for a couple > of hundred iterations. > > From my experience, the number of iterations to convergence in each > state/adjoint solve will decrease when increasing the (design) iterative > counter (i.e. the longer the process has gone on for) IF the initial guess is > the solution to the previous solve. This is because the design update is > smaller in the end of the process than in the beginning, and a smaller design > update leads to smaller changes in state/adjoint solution between subsequent > (design) iterations. This means that the numbers provided above are on the > low side: most likely the savings can be even more in the end of the design > process. > > Best regards, > Jonas Lundgren > > Från: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Skickat: den 21 augusti 2023 20:13 > Till: Jonas Lundgren <jonas.lundg...@liu.se <mailto:jonas.lundg...@liu.se>> > Kopia: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > Ämne: Re: [petsc-users] (Sub) KSP initial guess with PCREDISTRIBUTE > > > When you use 2 KSP so that you can use the previous "solution" as the > initial guess for the next problem, how much savings do you get? In > iterations inside PCREDISTRIBUTE and in time (relative to the entire linear > solver time and relative to the entire application run)? You can get this > information running with -log_view > > That is, run the 2 KSP simulation twice, once with the inner > KSPSetNonzeroInitialGuess() on and once with it off and compare the times for > the two cases. > > Thanks > > Barry > > Using KSPSetNonzeroInitialGuess() requires an extra matrix-vector product > and preconditioner application, so I would like to verify that you have a > measurable performance improvement with the initial guess. > > > > > > > > > On Aug 21, 2023, at 7:06 AM, Jonas Lundgren via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > > Dear PETSc users, > > I have a problem regarding the setting of initial guess to KSP when using > PCREDISTRIBUTE as the preconditioner. (The reason to why I use PCREDISTRIBUTE > is because I have a lot of fixed DOF in my problem, and PCREDISTRIBUTE > successfully reduces the problem size and therefore speeds up the solving). > > First, some details: > - I use a version of PETSc 3.19.1 > - The KSP I use is KSPPREONLY, as suggested in the manual pages of > PCREDISTRIBUTE:https://petsc.org/release/manualpages/PC/PCREDISTRIBUTE/ > - I use KSPBCGSL as sub-KSP. I can perfectly well solve my problem > using this as my main KSP, but the performance is much worse than when using > it as my sub-KSP (under KSPPREONLY+PCREDISTRIBUTE) due to the amount of fixed > DOF > - I am first solving a state problem, then an adjoint problem using > the same linear operator. > - The adjoint vector is used as sensitivity information to update a > design. After the design update, the state+adjoint problems are solved again > with a slightly updated linear operator. This is done for hundreds of > (design) iteration steps > - I want the initial guess for the state problem to be the state > solution from the previous (design) iteration, and same for the adjoint > problem > - I am aware of the default way of setting a custom initial guess: > KSPSetInitialGuessNonzero(ksp, PETSC_TRUE) together with providing the actual > guess in the x vector in the call to KSPSolve(ksp, b, x) > > The main problem is that PCREDISTRIBUTE internally doesn’t use the input > solution vector (x) when calling KSPSolve() for the sub-KSP. It zeroes out > the solution vector (x) when starting to build x = diag(A)^{-1} b in the > beginning of PCApply_Redistribute(), and uses red->x as the solution > vector/initial guess when calling KSPSolve(). Therefore, I cannot reach the > sub-KSP with an initial guess. > > Additionally, KSPPREONLY prohibits the use of having a nonzero initial guess > (the error message says “it doesn’t make sense”). I guess I can remove the > line raising this error and recompile the PETSc libraries, but it still won’t > solve the previously mentioned problem, which seems to be the hard nut to > crack. > > So far, I have found out that if I create 2 KSP object, one each for the > state and adjoint problems, it is enough with calling > KSPSetInitialGuessNonzero(subksp, PETSC_TRUE) on the subksp. It seems as if > the variable red->x in PCApply_Redistribute() is kept untouched in memory > between calls to the main KSP and therefore is used as (non-zero) initial > guess to the sub-KSP. This has been verified by introducing > PetscCall(PetscObjectCompose((PetscObject)pc,"redx",(PetscObject)red->x)); in > PCApply_Redistribute(), recompiling the PETSc library, and then inserting a > corresponding PetscObjectQuery((PetscObject)pc, "redx", (PetscObject > *)&redx); in my own program source file. > > However, I would like to only create 1 KSP to be used with both the state and > adjoint problems (same linear operator), for memory reasons. When I do this, > the initial guesses are mixed up between the two problems: the initial guess > for the adjoint problem is the solution to the state problem in the current > design iteration, and the initial guess for the state problem is the solution > to the adjoint problem in the previous design iteration. These are very bad > guesses that increases the time to convergence in each state/adjoint solve. > > So, the core of the problem (as far as I can understand) is that I want to > control the initial guess red->x in PCApply_Redistribute(). > > The only solution I can think of is to include a call to PetscObjectQuery() > in PCApply_Redistribute() to obtain a vector with the initial guess from my > main program. And then I need to keep track of the initial guesses for my > both problems in my main program myself (minor problem). This is maybe not > the neatest way, and I do not know if this approach affects the performance > negatively? Maybe one call each to PetscObjectQuery() and > PetscObjectCompose() per call to PCApply_Redistribute() is negligible? > > Is there another (and maybe simpler) solution to this problem? Maybe I can > SCATTER_FORWARD the input x vector in PCApply_Redistribute() before it is > zeroed out, together with allowing for non-zero initial guess in KSPPREONLY? > > Any help is welcome! > > > Best regards, > Jonas Lundgren
