On Tue, Mar 26, 2019 at 9:27 AM Myriam Peyrounette < myriam.peyroune...@idris.fr> wrote:
> I checked with -ksp_view (attached) but no prefix is associated with the > matrix. Some are associated to the KSP and PC, but none to the Mat > Another thing that could prevent options being used is that *SetFromOptions() is not called for the object. Thanks, Matt > Le 03/26/19 à 11:55, Dave May a écrit : > > > > On Tue, 26 Mar 2019 at 10:36, Myriam Peyrounette < > myriam.peyroune...@idris.fr> wrote: > >> Oh you were right, the three options are unsused (-matptap_via scalable, >> -inner_offdiag_matmatmult_via scalable and -inner_diag_matmatmult_via >> scalable). Does this mean I am not using the associated PtAP functions? >> > > No - not necessarily. All it means is the options were not parsed. > > If your matrices have an option prefix associated with them (e.g. abc) , > then you need to provide the option as > -abc_matptap_via scalable > > If you are not sure if you matrices have a prefix, look at the result of > -ksp_view (see below for an example) > > Mat Object: 2 MPI processes > > type: mpiaij > > rows=363, cols=363, bs=3 > > total: nonzeros=8649, allocated nonzeros=8649 > > total number of mallocs used during MatSetValues calls =0 > > Mat Object: (B_) 2 MPI processes > > type: mpiaij > > rows=363, cols=363, bs=3 > > total: nonzeros=8649, allocated nonzeros=8649 > > total number of mallocs used during MatSetValues calls =0 > > The first matrix has no options prefix, but the second does and it's > called "B_". > > > > > >> Myriam >> >> Le 03/26/19 à 11:10, Dave May a écrit : >> >> >> On Tue, 26 Mar 2019 at 09:52, Myriam Peyrounette via petsc-users < >> petsc-users@mcs.anl.gov> wrote: >> >>> How can I be sure they are indeed used? Can I print this information in >>> some log file? >>> >> Yes. Re-run the job with the command line option >> >> -options_left true >> >> This will report all options parsed, and importantly, will also indicate >> if any options were unused. >> >> >> Thanks >> Dave >> >> Thanks in advance >>> >>> Myriam >>> >>> Le 03/25/19 à 18:24, Matthew Knepley a écrit : >>> >>> On Mon, Mar 25, 2019 at 10:54 AM Myriam Peyrounette via petsc-users < >>> petsc-users@mcs.anl.gov> wrote: >>> >>>> Hi, >>>> >>>> thanks for the explanations. I tried the last PETSc version (commit >>>> fbc5705bc518d02a4999f188aad4ccff5f754cbf), which includes the patch you >>>> talked about. But the memory scaling shows no improvement (see scaling >>>> attached), even when using the "scalable" options :( >>>> >>>> I had a look at the PETSc functions MatPtAPNumeric_MPIAIJ_MPIAIJ and >>>> MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the differences before and >>>> after the first "bad" commit), but I can't find what induced this memory >>>> issue. >>>> >>> Are you sure that the option was used? It just looks suspicious to me >>> that they use exactly the same amount of memory. It should be different, >>> even if it does not solve the problem. >>> >>> Thanks, >>> >>> Matt >>> >>>> Myriam >>>> >>>> >>>> >>>> >>>> Le 03/20/19 à 17:38, Fande Kong a écrit : >>>> >>>> Hi Myriam, >>>> >>>> There are three algorithms in PETSc to do PtAP ( const char >>>> *algTypes[3] = {"scalable","nonscalable","hypre"};), and can be specified >>>> using the petsc options: -matptap_via xxxx. >>>> >>>> (1) -matptap_via hypre: This call the hypre package to do the PtAP >>>> trough an all-at-once triple product. In our experiences, it is the most >>>> memory efficient, but could be slow. >>>> >>>> (2) -matptap_via scalable: This involves a row-wise algorithm plus an >>>> outer product. This will use more memory than hypre, but way faster. This >>>> used to have a bug that could take all your memory, and I have a fix at >>>> https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff. >>>> When using this option, we may want to have extra options such as >>>> -inner_offdiag_matmatmult_via scalable -inner_diag_matmatmult_via >>>> scalable to select inner scalable algorithms. >>>> >>>> (3) -matptap_via nonscalable: Suppose to be even faster, but use more >>>> memory. It does dense matrix operations. >>>> >>>> >>>> Thanks, >>>> >>>> Fande Kong >>>> >>>> >>>> >>>> >>>> On Wed, Mar 20, 2019 at 10:06 AM Myriam Peyrounette via petsc-users < >>>> petsc-users@mcs.anl.gov> wrote: >>>> >>>>> More precisely: something happens when upgrading the functions >>>>> MatPtAPNumeric_MPIAIJ_MPIAIJ and/or MatPtAPSymbolic_MPIAIJ_MPIAIJ. >>>>> >>>>> Unfortunately, there are a lot of differences between the old and new >>>>> versions of these functions. I keep investigating but if you have any >>>>> idea, >>>>> please let me know. >>>>> >>>>> Best, >>>>> >>>>> Myriam >>>>> >>>>> Le 03/20/19 à 13:48, Myriam Peyrounette a écrit : >>>>> >>>>> Hi all, >>>>> >>>>> I used git bisect to determine when the memory need increased. I found >>>>> that the first "bad" commit is aa690a28a7284adb519c28cb44eae20a2c131c85. >>>>> >>>>> Barry was right, this commit seems to be about an evolution of >>>>> MatPtAPSymbolic_MPIAIJ_MPIAIJ. >>>>> You mentioned the option "-matptap_via scalable" but I can't find any >>>>> information about it. Can you tell me more? >>>>> >>>>> Thanks >>>>> >>>>> Myriam >>>>> >>>>> >>>>> Le 03/11/19 à 14:40, Mark Adams a écrit : >>>>> >>>>> Is there a difference in memory usage on your tiny problem? I assume >>>>> no. >>>>> >>>>> I don't see anything that could come from GAMG other than the RAP >>>>> stuff that you have discussed already. >>>>> >>>>> On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette < >>>>> myriam.peyroune...@idris.fr> wrote: >>>>> >>>>>> The code I am using here is the example 42 of PETSc ( >>>>>> https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html). >>>>>> Indeed it solves the Stokes equation. I thought it was a good idea to use >>>>>> an example you might know (and didn't find any that uses GAMG >>>>>> functions). I >>>>>> just changed the PCMG setup so that the memory problem appears. And it >>>>>> appears when adding PCGAMG. >>>>>> >>>>>> I don't care about the performance or even the result rightness here, >>>>>> but only about the difference in memory use between 3.6 and 3.10. Do you >>>>>> think finding a more adapted script would help? >>>>>> >>>>>> I used the threshold of 0.1 only once, at the beginning, to test its >>>>>> influence. I used the default threshold (of 0, I guess) for all the other >>>>>> runs. >>>>>> >>>>>> Myriam >>>>>> >>>>>> Le 03/11/19 à 13:52, Mark Adams a écrit : >>>>>> >>>>>> In looking at this larger scale run ... >>>>>> >>>>>> * Your eigen estimates are much lower than your tiny test problem. >>>>>> But this is Stokes apparently and it should not work anyway. Maybe you >>>>>> have >>>>>> a small time step that adds a lot of mass that brings the eigen estimates >>>>>> down. And your min eigenvalue (not used) is positive. I would expect >>>>>> negative for Stokes ... >>>>>> >>>>>> * You seem to be setting a threshold value of 0.1 -- that is very high >>>>>> >>>>>> * v3.6 says "using nonzero initial guess" but this is not in v3.10. >>>>>> Maybe we just stopped printing that. >>>>>> >>>>>> * There were some changes to coasening parameters in going from v3.6 >>>>>> but it does not look like your problem was effected. (The coarsening algo >>>>>> is non-deterministic by default and you can see small difference on >>>>>> different runs) >>>>>> >>>>>> * We may have also added a "noisy" RHS for eigen estimates by default >>>>>> from v3.6. >>>>>> >>>>>> * And for non-symetric problems you can try -pc_gamg_agg_nsmooths 0, >>>>>> but again GAMG is not built for Stokes anyway. >>>>>> >>>>>> >>>>>> On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette < >>>>>> myriam.peyroune...@idris.fr> wrote: >>>>>> >>>>>>> I used PCView to display the size of the linear system in each level >>>>>>> of the MG. You'll find the outputs attached to this mail (zip file) for >>>>>>> both the default threshold value and a value of 0.1, and for both 3.6 >>>>>>> and >>>>>>> 3.10 PETSc versions. >>>>>>> >>>>>>> For convenience, I summarized the information in a graph, also >>>>>>> attached (png file). >>>>>>> >>>>>>> As you can see, there are slight differences between the two >>>>>>> versions but none is critical, in my opinion. Do you see anything >>>>>>> suspicious in the outputs? >>>>>>> >>>>>>> + I can't find the default threshold value. Do you know where I can >>>>>>> find it? >>>>>>> >>>>>>> Thanks for the follow-up >>>>>>> >>>>>>> Myriam >>>>>>> >>>>>>> Le 03/05/19 à 14:06, Matthew Knepley a écrit : >>>>>>> >>>>>>> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette < >>>>>>> myriam.peyroune...@idris.fr> wrote: >>>>>>> >>>>>>>> Hi Matt, >>>>>>>> >>>>>>>> I plotted the memory scalings using different threshold values. The >>>>>>>> two scalings are slightly translated (from -22 to -88 mB) but this >>>>>>>> gain is >>>>>>>> neglectable. The 3.6-scaling keeps being robust while the 3.10-scaling >>>>>>>> deteriorates. >>>>>>>> >>>>>>>> Do you have any other suggestion? >>>>>>>> >>>>>>> Mark, what is the option she can give to output all the GAMG data? >>>>>>> >>>>>>> Also, run using -ksp_view. GAMG will report all the sizes of its >>>>>>> grids, so it should be easy to see >>>>>>> if the coarse grid sizes are increasing, and also what the effect of >>>>>>> the threshold value is. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> Thanks >>>>>>>> Myriam >>>>>>>> >>>>>>>> Le 03/02/19 à 02:27, Matthew Knepley a écrit : >>>>>>>> >>>>>>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users < >>>>>>>> petsc-users@mcs.anl.gov> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc >>>>>>>>> version >>>>>>>>> to 3.10, this code has a bad memory scaling. >>>>>>>>> >>>>>>>>> To report this issue, I took the PETSc script ex42.c and slightly >>>>>>>>> modified it so that the KSP and PC configurations are the same as >>>>>>>>> in my >>>>>>>>> code. In particular, I use a "personnalised" multi-grid method. The >>>>>>>>> modifications are indicated by the keyword "TopBridge" in the >>>>>>>>> attached >>>>>>>>> scripts. >>>>>>>>> >>>>>>>>> To plot the memory (weak) scaling, I ran four calculations for each >>>>>>>>> script with increasing problem sizes and computations cores: >>>>>>>>> >>>>>>>>> 1. 100,000 elts on 4 cores >>>>>>>>> 2. 1 million elts on 40 cores >>>>>>>>> 3. 10 millions elts on 400 cores >>>>>>>>> 4. 100 millions elts on 4,000 cores >>>>>>>>> >>>>>>>>> The resulting graph is also attached. The scaling using PETSc 3.10 >>>>>>>>> clearly deteriorates for large cases, while the one using PETSc >>>>>>>>> 3.6 is >>>>>>>>> robust. >>>>>>>>> >>>>>>>>> After a few tests, I found that the scaling is mostly sensitive to >>>>>>>>> the >>>>>>>>> use of the AMG method for the coarse grid (line 1780 in >>>>>>>>> main_ex42_petsc36.cc). In particular, the performance strongly >>>>>>>>> deteriorates when commenting lines 1777 to 1790 (in >>>>>>>>> main_ex42_petsc36.cc). >>>>>>>>> >>>>>>>>> Do you have any idea of what changed between version 3.6 and >>>>>>>>> version >>>>>>>>> 3.10 that may imply such degradation? >>>>>>>>> >>>>>>>> >>>>>>>> I believe the default values for PCGAMG changed between versions. >>>>>>>> It sounds like the coarsening rate >>>>>>>> is not great enough, so that these grids are too large. This can be >>>>>>>> set using: >>>>>>>> >>>>>>>> >>>>>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >>>>>>>> >>>>>>>> There is some explanation of this effect on that page. Let us know >>>>>>>> if setting this does not correct the situation. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Let me know if you need further information. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> Myriam Peyrounette >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Myriam Peyrounette >>>>>>>>> CNRS/IDRIS - HLST >>>>>>>>> -- >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Myriam Peyrounette >>>>>>>> CNRS/IDRIS - HLST >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Myriam Peyrounette >>>>>>> CNRS/IDRIS - HLST >>>>>>> -- >>>>>>> >>>>>>> >>>>>> -- >>>>>> Myriam Peyrounette >>>>>> CNRS/IDRIS - HLST >>>>>> -- >>>>>> >>>>>> >>>>> -- >>>>> Myriam Peyrounette >>>>> CNRS/IDRIS - HLST >>>>> -- >>>>> >>>>> >>>>> -- >>>>> Myriam Peyrounette >>>>> CNRS/IDRIS - HLST >>>>> -- >>>>> >>>>> >>>> -- >>>> Myriam Peyrounette >>>> CNRS/IDRIS - HLST >>>> -- >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/%7Eknepley/> >>> >>> >>> -- >>> Myriam Peyrounette >>> CNRS/IDRIS - HLST >>> -- >>> >>> >> -- >> Myriam Peyrounette >> CNRS/IDRIS - HLST >> -- >> >> > -- > Myriam Peyrounette > CNRS/IDRIS - HLST > -- > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>