> On 26 Nov 2025, at 10:26 PM, Randall Mackie <[email protected]> wrote:
>
> Hi Pierre,
>
> Following up on this email, what are good options to use when trying PCHPDDM
> for the first time?
There is no single answer to this question, especially when solving systems in
a purely algebraic manner.
Some examples can be found in the repository:
$ git grep " hpddm" "*/ex*"
> For example, what did you use here that worked so well?
What worked well here may totally fail for other problems, so unless you are
solving the same problem, then the following may not work, but here is what I
get with -options_view.
Thanks,
Pierre
Linear A_ solve converged due to CONVERGED_RTOL iterations 5
Linear B_ solve converged due to CONVERGED_RTOL iterations 39
#PETSc Option Table entries:
-A_ksp_converged_reason # (source: command line)
-A_ksp_type fgmres # (source: command line)
-A_pc_hpddm_coarse_mat_type baij # (source: command line)
-A_pc_hpddm_harmonic_overlap 2 # (source: command line)
-A_pc_hpddm_levels_1_sub_pc_type lu # (source: command line)
-A_pc_hpddm_levels_1_svd_nsv 120 # (source: command line)
-A_pc_type hpddm # (source: command line)
-B_ksp_converged_reason # (source: command line)
-B_ksp_type fgmres # (source: command line)
-B_pc_hpddm_coarse_mat_type baij # (source: command line)
-B_pc_hpddm_harmonic_overlap 2 # (source: command line)
-B_pc_hpddm_levels_1_sub_pc_type lu # (source: command line)
-B_pc_hpddm_levels_1_svd_nsv 120 # (source: command line)
-B_pc_type hpddm # (source: command line)
-matload_block_size 1 # (source: file)
-options_view # (source: command line)
-vecload_block_size 1 # (source: file)
#End of PETSc Option Table entries
> Thanks,
>
> Randy M.
>
>
>
>> On Oct 24, 2025, at 8:14 AM, Pierre Jolivet <[email protected]> wrote:
>>
>>
>>> On 24 Oct 2025, at 4:38 PM, Nils Schween <[email protected]> wrote:
>>>
>>> Thank you very much Pierre!
>>>
>>> I was not aware of the fact that the fill-in in the ILU decides about
>>> its quality. But its clear now. I will just test what level of fill we
>>> need for our application.
>>
>> I’ll note that block Jacobi and ILU are not very efficient solvers in most
>> instances.
>> I tried much fancier algebraic preconditioners such as BoomerAMG and GAMG on
>> your problem, and they are failing hard out-of-the-box.
>> Without knowing much more on the problem, it’s difficult to setup.
>> We also have other more robust preconditioners in PETSc by means of domain
>> decomposition methods.
>> deal.II is interfaced with PCBDDC (which is also somewhat difficult to tune
>> in a fully algebraic mode) and you could also use PCHPDDM (in fully
>> algebraic mode).
>> On this toy problem, PCHPDDM performs much better in terms of iteration than
>> the simpler PCBJACOBI + (sub) PCILU.
>> Of course, as we always advise our users, it’s best to do a little bit of
>> literature survey to find the best method for your application, I doubt it’s
>> PCBJACOBI.
>> If the solver part is not a problem in your application, just carry on with
>> what’s easiest for you.
>> If you want some precise help on either PCBDDC or PCHPDDM, feel free to get
>> in touch with me in private.
>>
>> Thanks,
>> Pierre
>>
>> PCGAMG
>> Linear A_ solve did not converge due to DIVERGED_ITS iterations 1000
>> Linear B_ solve did not converge due to DIVERGED_ITS iterations 1000
>> PCHYPRE
>> Linear A_ solve did not converge due to DIVERGED_NANORINF iterations 0
>> Linear B_ solve did not converge due to DIVERGED_NANORINF iterations 0
>> PCHPDDM
>> Linear A_ solve converged due to CONVERGED_RTOL iterations 4
>> Linear B_ solve converged due to CONVERGED_RTOL iterations 38
>> PCBJACOBI
>> Linear A_ solve converged due to CONVERGED_RTOL iterations 134
>> Linear B_ solve did not converge due to DIVERGED_ITS iterations 1000
>>
>>> Once more thanks,
>>> Nils
>>>
>>>
>>> Pierre Jolivet <[email protected]> writes:
>>>
>>>>> On 24 Oct 2025, at 1:52 PM, Nils Schween <[email protected]>
>>>>> wrote:
>>>>>
>>>>> Dear PETSc users, Dear PETSc developers,
>>>>>
>>>>> in our software we are solving a linear system with PETSc using GMRES
>>>>> in conjunction with a BlockJacobi preconditioner, i.e. the default of
>>>>> the KSP object.
>>>>>
>>>>> We have two versions of the system matrix, say A and B. The difference
>>>>> between them is the non-zero pattern. The non-zero pattern of matrix B
>>>>> is a subset of the one of matrix A. Their values should be identical.
>>>>>
>>>>> We solve the linear system, using A yields a solution after some
>>>>> iterations, whereas using B does not converge.
>>>>>
>>>>> I created binary files of the two matrices, the right-hand side, and
>>>>> wrote a small PETSc programm, which loads them and demonstrates the
>>>>> issue. I attach the files to this email.
>>>>>
>>>>> We would like to understand why the solver-preconditioner combination
>>>>> works in case A and not in case B. Can you help us finding this out?
>>>>>
>>>>> To test if the two matrices are identical, I substracted them and
>>>>> computed the Frobenius norm of the result. It is zero.
>>>>
>>>> The default subdomain solver is ILU(0).
>>>> By definition, this won’t allow fill-in.
>>>> So when you are not storing the zeros in B, the quality of your PC is much
>>>> worse.
>>>> You can check this yourself with -A_ksp_view -B_ksp_view:
>>>> […]
>>>> 0 levels of fill
>>>> tolerance for zero pivot 2.22045e-14
>>>> matrix ordering: natural
>>>> factor fill ratio given 1., needed 1.
>>>> Factored matrix follows:
>>>> Mat Object: (A_) 1 MPI process
>>>> type: seqaij
>>>> rows=1664, cols=1664
>>>> package used to perform factorization: petsc
>>>> total: nonzeros=117760, allocated nonzeros=117760
>>>> using I-node routines: found 416 nodes, limit used is 5
>>>> […]
>>>> 0 levels of fill
>>>> tolerance for zero pivot 2.22045e-14
>>>> matrix ordering: natural
>>>> factor fill ratio given 1., needed 1.
>>>> Factored matrix follows:
>>>> Mat Object: (B_) 1 MPI process
>>>> type: seqaij
>>>> rows=1664, cols=1664
>>>> package used to perform factorization: petsc
>>>> total: nonzeros=49408, allocated nonzeros=49408
>>>> not using I-node routines
>>>>
>>>> Check the number of nonzeros of both factored Mat.
>>>> With -B_pc_factor_levels 3, you’ll get roughly similar convergence speed
>>>> (and density in the factored Mat of both PC).
>>>>
>>>> Thanks,
>>>> Pierre
>>>>
>>>>>
>>>>> To give you more context, we solve a system of partial differential
>>>>> equations that models astrophysical plasmas. It is essentially a system
>>>>> of advection-reaction equations. We use a discontinuous Galerkin (dG)
>>>>> method. Our code relies on the finite element library library deal.ii
>>>>> and its PETSc interface. The system matrices A and B are the result of
>>>>> the (dG) discretisation. We GMRES with a BlockJaboci preconditioner,
>>>>> because we do not know any better.
>>>>>
>>>>> I tested the code I sent with PETSc 3.24.0 and 3.19.1 on my workstation,
>>>>> i.e.
>>>>> Linux home-desktop 6.17.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 12 Oct 2025
>>>>> 12:45:18 +0000 x86_64 GNU/Linux
>>>>> I use OpenMPI 5.0.8 and I compiled with mpicc, which in my cases use
>>>>> gcc.
>>>>>
>>>>> In case you need more information. Please let me know.
>>>>> Any help is appreciated.
>>>>>
>>>>> Thank you,
>>>>> Nils
>>>>> <example.tar.gz>
>>>>>
>>>>> --
>>>>> Nils Schween
>>>>>
>>>>> Phone: +49 6221 516 557
>>>>> Mail: [email protected]
>>>>> PGP-Key: 4DD3DCC0532EE96DB0C1F8B5368DBFA14CB81849
>>>>>
>>>>> Max Planck Institute for Nuclear Physics
>>>>> Astrophysical Plasma Theory (APT)
>>>>> Saupfercheckweg 1, D-69117 Heidelberg
>>>>> https://urldefense.us/v3/__https://www.mpi-hd.mpg.de/mpi/en/research/scientific-divisions-and-groups/independent-research-groups/apt__;!!G_uCfscf7eWS!YoOlZjX4v-hbz0Oaawvh2Yy3nCbpHcafn1VjON06Or7f-WVrzGzD9SMcky5YJAyzVu62BIfzC5cpshSkkkpKpA$
>>>
>>> --
>>> Nils Schween
>>> PhD Student
>>>
>>> Phone: +49 6221 516 557
>>> Mail: [email protected]
>>> PGP-Key: 4DD3DCC0532EE96DB0C1F8B5368DBFA14CB81849
>>>
>>> Max Planck Institute for Nuclear Physics
>>> Astrophysical Plasma Theory (APT)
>>> Saupfercheckweg 1, D-69117 Heidelberg
>>> https://urldefense.us/v3/__https://www.mpi-hd.mpg.de/mpi/en/research/scientific-divisions-and-groups/independent-research-groups/apt__;!!G_uCfscf7eWS!YoOlZjX4v-hbz0Oaawvh2Yy3nCbpHcafn1VjON06Or7f-WVrzGzD9SMcky5YJAyzVu62BIfzC5cpshSkkkpKpA$
>>
>
