Hi, thanks for the explanations. I tried the last PETSc version (commit fbc5705bc518d02a4999f188aad4ccff5f754cbf), which includes the patch you talked about. But the memory scaling shows no improvement (see scaling attached), even when using the "scalable" options :(
I had a look at the PETSc functions MatPtAPNumeric_MPIAIJ_MPIAIJ and MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the differences before and after the first "bad" commit), but I can't find what induced this memory issue. Myriam Le 03/20/19 à 17:38, Fande Kong a écrit : > Hi Myriam, > > There are three algorithms in PETSc to do PtAP ( const char > *algTypes[3] = {"scalable","nonscalable","hypre"};), and can be > specified using the petsc options: -matptap_via xxxx. > > (1) -matptap_via hypre: This call the hypre package to do the PtAP > trough an all-at-once triple product. In our experiences, it is the > most memory efficient, but could be slow. > > (2) -matptap_via scalable: This involves a row-wise algorithm plus an > outer product. This will use more memory than hypre, but way faster. > This used to have a bug that could take all your memory, and I have a > fix > at > https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff. > > When using this option, we may want to have extra options such as > -inner_offdiag_matmatmult_via scalable -inner_diag_matmatmult_via > scalable to select inner scalable algorithms. > > (3) -matptap_via nonscalable: Suppose to be even faster, but use > more memory. It does dense matrix operations. > > > Thanks, > > Fande Kong > > > > > On Wed, Mar 20, 2019 at 10:06 AM Myriam Peyrounette via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > > More precisely: something happens when upgrading the functions > MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. > > Unfortunately, there are a lot of differences between the old and > new versions of these functions. I keep investigating but if you > have any idea, please let me know. > > Best, > > Myriam > > > Le 03/20/19 à 13:48, Myriam Peyrounette a écrit : >> >> Hi all, >> >> I used git bisect to determine when the memory need increased. I >> found that the first "bad" commit is >> aa690a28a7284adb519c28cb44eae20a2c131c85. >> >> Barry was right, this commit seems to be about an evolution of >> MatPtAPSymbolic_MPIAIJ_MPIAIJ. You mentioned the option >> "-matptap_via scalable" but I can't find any information about >> it. Can you tell me more? >> >> Thanks >> >> Myriam >> >> >> Le 03/11/19 à 14:40, Mark Adams a écrit : >>> Is there a difference in memory usage on your tiny problem? I >>> assume no. >>> >>> I don't see anything that could come from GAMG other than the >>> RAP stuff that you have discussed already. >>> >>> On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette >>> <myriam.peyroune...@idris.fr >>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>> >>> The code I am using here is the example 42 of PETSc >>> >>> (https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html). >>> Indeed it solves the Stokes equation. I thought it was a >>> good idea to use an example you might know (and didn't find >>> any that uses GAMG functions). I just changed the PCMG setup >>> so that the memory problem appears. And it appears when >>> adding PCGAMG. >>> >>> I don't care about the performance or even the result >>> rightness here, but only about the difference in memory use >>> between 3.6 and 3.10. Do you think finding a more adapted >>> script would help? >>> >>> I used the threshold of 0.1 only once, at the beginning, to >>> test its influence. I used the default threshold (of 0, I >>> guess) for all the other runs. >>> >>> Myriam >>> >>> >>> Le 03/11/19 à 13:52, Mark Adams a écrit : >>>> In looking at this larger scale run ... >>>> >>>> * Your eigen estimates are much lower than your tiny test >>>> problem. But this is Stokes apparently and it should not >>>> work anyway. Maybe you have a small time step that adds a >>>> lot of mass that brings the eigen estimates down. And your >>>> min eigenvalue (not used) is positive. I would expect >>>> negative for Stokes ... >>>> >>>> * You seem to be setting a threshold value of 0.1 -- that >>>> is very high >>>> >>>> * v3.6 says "using nonzero initial guess" but this is not >>>> in v3.10. Maybe we just stopped printing that. >>>> >>>> * There were some changes to coasening parameters in going >>>> from v3.6 but it does not look like your problem was >>>> effected. (The coarsening algo is non-deterministic by >>>> default and you can see small difference on different runs) >>>> >>>> * We may have also added a "noisy" RHS for eigen estimates >>>> by default from v3.6. >>>> >>>> * And for non-symetric problems you can try >>>> -pc_gamg_agg_nsmooths 0, but again GAMG is not built for >>>> Stokes anyway. >>>> >>>> >>>> On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette >>>> <myriam.peyroune...@idris.fr >>>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>>> >>>> I used PCView to display the size of the linear system >>>> in each level of the MG. You'll find the outputs >>>> attached to this mail (zip file) for both the default >>>> threshold value and a value of 0.1, and for both 3.6 >>>> and 3.10 PETSc versions. >>>> >>>> For convenience, I summarized the information in a >>>> graph, also attached (png file). >>>> >>>> As you can see, there are slight differences between >>>> the two versions but none is critical, in my opinion. >>>> Do you see anything suspicious in the outputs? >>>> >>>> + I can't find the default threshold value. Do you know >>>> where I can find it? >>>> >>>> Thanks for the follow-up >>>> >>>> Myriam >>>> >>>> >>>> Le 03/05/19 à 14:06, Matthew Knepley a écrit : >>>>> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette >>>>> <myriam.peyroune...@idris.fr >>>>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>>>> >>>>> Hi Matt, >>>>> >>>>> I plotted the memory scalings using different >>>>> threshold values. The two scalings are slightly >>>>> translated (from -22 to -88 mB) but this gain is >>>>> neglectable. The 3.6-scaling keeps being robust >>>>> while the 3.10-scaling deteriorates. >>>>> >>>>> Do you have any other suggestion? >>>>> >>>>> Mark, what is the option she can give to output all >>>>> the GAMG data? >>>>> >>>>> Also, run using -ksp_view. GAMG will report all the >>>>> sizes of its grids, so it should be easy to see >>>>> if the coarse grid sizes are increasing, and also what >>>>> the effect of the threshold value is. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> Thanks >>>>> >>>>> Myriam >>>>> >>>>> Le 03/02/19 à 02:27, Matthew Knepley a écrit : >>>>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam >>>>>> Peyrounette via petsc-users >>>>>> <petsc-users@mcs.anl.gov >>>>>> <mailto:petsc-users@mcs.anl.gov>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I used to run my code with PETSc 3.6. Since I >>>>>> upgraded the PETSc version >>>>>> to 3.10, this code has a bad memory scaling. >>>>>> >>>>>> To report this issue, I took the PETSc script >>>>>> ex42.c and slightly >>>>>> modified it so that the KSP and PC >>>>>> configurations are the same as in my >>>>>> code. In particular, I use a "personnalised" >>>>>> multi-grid method. The >>>>>> modifications are indicated by the keyword >>>>>> "TopBridge" in the attached >>>>>> scripts. >>>>>> >>>>>> To plot the memory (weak) scaling, I ran four >>>>>> calculations for each >>>>>> script with increasing problem sizes and >>>>>> computations cores: >>>>>> >>>>>> 1. 100,000 elts on 4 cores >>>>>> 2. 1 million elts on 40 cores >>>>>> 3. 10 millions elts on 400 cores >>>>>> 4. 100 millions elts on 4,000 cores >>>>>> >>>>>> The resulting graph is also attached. The >>>>>> scaling using PETSc 3.10 >>>>>> clearly deteriorates for large cases, while >>>>>> the one using PETSc 3.6 is >>>>>> robust. >>>>>> >>>>>> After a few tests, I found that the scaling >>>>>> is mostly sensitive to the >>>>>> use of the AMG method for the coarse grid >>>>>> (line 1780 in >>>>>> main_ex42_petsc36.cc). In particular, the >>>>>> performance strongly >>>>>> deteriorates when commenting lines 1777 to >>>>>> 1790 (in main_ex42_petsc36.cc). >>>>>> >>>>>> Do you have any idea of what changed between >>>>>> version 3.6 and version >>>>>> 3.10 that may imply such degradation? >>>>>> >>>>>> >>>>>> I believe the default values for PCGAMG changed >>>>>> between versions. It sounds like the coarsening rate >>>>>> is not great enough, so that these grids are too >>>>>> large. This can be set using: >>>>>> >>>>>> >>>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >>>>>> >>>>>> There is some explanation of this effect on that >>>>>> page. Let us know if setting this does not >>>>>> correct the situation. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>> Let me know if you need further information. >>>>>> >>>>>> Best, >>>>>> >>>>>> Myriam Peyrounette >>>>>> >>>>>> >>>>>> -- >>>>>> Myriam Peyrounette >>>>>> CNRS/IDRIS - HLST >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before >>>>>> they begin their experiments is infinitely more >>>>>> interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>>> >>>>> -- >>>>> Myriam Peyrounette >>>>> CNRS/IDRIS - HLST >>>>> -- >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they >>>>> begin their experiments is infinitely more interesting >>>>> than any results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>> >>>> -- >>>> Myriam Peyrounette >>>> CNRS/IDRIS - HLST >>>> -- >>>> >>> >>> -- >>> Myriam Peyrounette >>> CNRS/IDRIS - HLST >>> -- >>> >> >> -- >> Myriam Peyrounette >> CNRS/IDRIS - HLST >> -- > > -- > Myriam Peyrounette > CNRS/IDRIS - HLST > -- > -- Myriam Peyrounette CNRS/IDRIS - HLST --
smime.p7s
Description: Signature cryptographique S/MIME