Oh you were right, the three options are unsused (-matptap_via scalable, -inner_offdiag_matmatmult_via scalable and -inner_diag_matmatmult_via scalable). Does this mean I am not using the associated PtAP functions?
Myriam Le 03/26/19 à 11:10, Dave May a écrit : > > On Tue, 26 Mar 2019 at 09:52, Myriam Peyrounette via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > > How can I be sure they are indeed used? Can I print this > information in some log file? > > Yes. Re-run the job with the command line option > > -options_left true > > This will report all options parsed, and importantly, will also > indicate if any options were unused. > > > Thanks > Dave > > Thanks in advance > > Myriam > > > Le 03/25/19 à 18:24, Matthew Knepley a écrit : >> On Mon, Mar 25, 2019 at 10:54 AM Myriam Peyrounette via >> petsc-users <petsc-users@mcs.anl.gov >> <mailto:petsc-users@mcs.anl.gov>> wrote: >> >> Hi, >> >> thanks for the explanations. I tried the last PETSc version >> (commit fbc5705bc518d02a4999f188aad4ccff5f754cbf), which >> includes the patch you talked about. But the memory scaling >> shows no improvement (see scaling attached), even when using >> the "scalable" options :( >> >> I had a look at the PETSc functions >> MatPtAPNumeric_MPIAIJ_MPIAIJ and >> MatPtAPSymbolic_MPIAIJ_MPIAIJ (especially at the differences >> before and after the first "bad" commit), but I can't find >> what induced this memory issue. >> >> Are you sure that the option was used? It just looks suspicious >> to me that they use exactly the same amount of memory. It should >> be different, even if it does not solve the problem. >> >> Thanks, >> >> Matt >> >> Myriam >> >> >> >> >> Le 03/20/19 à 17:38, Fande Kong a écrit : >>> Hi Myriam, >>> >>> There are three algorithms in PETSc to do PtAP ( const char >>> *algTypes[3] = {"scalable","nonscalable","hypre"};), >>> and can be specified using the petsc options: -matptap_via xxxx. >>> >>> (1) -matptap_via hypre: This call the hypre package to do >>> the PtAP trough an all-at-once triple product. In our >>> experiences, it is the most memory efficient, but could be slow. >>> >>> (2) -matptap_via scalable: This involves a row-wise >>> algorithm plus an outer product. This will use more memory >>> than hypre, but way faster. This used to have a bug that >>> could take all your memory, and I have a fix >>> at >>> https://bitbucket.org/petsc/petsc/pull-requests/1452/mpiptap-enable-large-scale-simulations/diff. >>> >>> When using this option, we may want to have extra options >>> such as -inner_offdiag_matmatmult_via scalable >>> -inner_diag_matmatmult_via scalable to select inner >>> scalable algorithms. >>> >>> (3) -matptap_via nonscalable: Suppose to be even faster, >>> but use more memory. It does dense matrix operations. >>> >>> >>> Thanks, >>> >>> Fande Kong >>> >>> >>> >>> >>> On Wed, Mar 20, 2019 at 10:06 AM Myriam Peyrounette via >>> petsc-users <petsc-users@mcs.anl.gov >>> <mailto:petsc-users@mcs.anl.gov>> wrote: >>> >>> More precisely: something happens when upgrading the >>> functions MatPtAPNumeric_MPIAIJ_MPIAIJ and/or >>> MatPtAPSymbolic_MPIAIJ_MPIAIJ. >>> >>> Unfortunately, there are a lot of differences between >>> the old and new versions of these functions. I keep >>> investigating but if you have any idea, please let me know. >>> >>> Best, >>> >>> Myriam >>> >>> >>> Le 03/20/19 à 13:48, Myriam Peyrounette a écrit : >>>> >>>> Hi all, >>>> >>>> I used git bisect to determine when the memory need >>>> increased. I found that the first "bad" commit is >>>> aa690a28a7284adb519c28cb44eae20a2c131c85. >>>> >>>> Barry was right, this commit seems to be about an >>>> evolution of MatPtAPSymbolic_MPIAIJ_MPIAIJ. You >>>> mentioned the option "-matptap_via scalable" but I >>>> can't find any information about it. Can you tell me more? >>>> >>>> Thanks >>>> >>>> Myriam >>>> >>>> >>>> Le 03/11/19 à 14:40, Mark Adams a écrit : >>>>> Is there a difference in memory usage on your tiny >>>>> problem? I assume no. >>>>> >>>>> I don't see anything that could come from GAMG other >>>>> than the RAP stuff that you have discussed already. >>>>> >>>>> On Mon, Mar 11, 2019 at 9:32 AM Myriam Peyrounette >>>>> <myriam.peyroune...@idris.fr >>>>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>>>> >>>>> The code I am using here is the example 42 of >>>>> PETSc >>>>> >>>>> (https://www.mcs.anl.gov/petsc/petsc-3.9/src/ksp/ksp/examples/tutorials/ex42.c.html). >>>>> Indeed it solves the Stokes equation. I thought it >>>>> was a good idea to use an example you might know >>>>> (and didn't find any that uses GAMG functions). I >>>>> just changed the PCMG setup so that the memory >>>>> problem appears. And it appears when adding PCGAMG. >>>>> >>>>> I don't care about the performance or even the >>>>> result rightness here, but only about the >>>>> difference in memory use between 3.6 and 3.10. Do >>>>> you think finding a more adapted script would help? >>>>> >>>>> I used the threshold of 0.1 only once, at the >>>>> beginning, to test its influence. I used the >>>>> default threshold (of 0, I guess) for all the >>>>> other runs. >>>>> >>>>> Myriam >>>>> >>>>> >>>>> Le 03/11/19 à 13:52, Mark Adams a écrit : >>>>>> In looking at this larger scale run ... >>>>>> >>>>>> * Your eigen estimates are much lower than your >>>>>> tiny test problem. But this is Stokes apparently >>>>>> and it should not work anyway. Maybe you have a >>>>>> small time step that adds a lot of mass that >>>>>> brings the eigen estimates down. And your min >>>>>> eigenvalue (not used) is positive. I would expect >>>>>> negative for Stokes ... >>>>>> >>>>>> * You seem to be setting a threshold value of 0.1 >>>>>> -- that is very high >>>>>> >>>>>> * v3.6 says "using nonzero initial guess" but >>>>>> this is not in v3.10. Maybe we just stopped >>>>>> printing that. >>>>>> >>>>>> * There were some changes to coasening parameters >>>>>> in going from v3.6 but it does not look like your >>>>>> problem was effected. (The coarsening algo is >>>>>> non-deterministic by default and you can see >>>>>> small difference on different runs) >>>>>> >>>>>> * We may have also added a "noisy" RHS for eigen >>>>>> estimates by default from v3.6. >>>>>> >>>>>> * And for non-symetric problems you can try >>>>>> -pc_gamg_agg_nsmooths 0, but again GAMG is not >>>>>> built for Stokes anyway. >>>>>> >>>>>> >>>>>> On Tue, Mar 5, 2019 at 11:53 AM Myriam >>>>>> Peyrounette <myriam.peyroune...@idris.fr >>>>>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>>>>> >>>>>> I used PCView to display the size of the >>>>>> linear system in each level of the MG. You'll >>>>>> find the outputs attached to this mail (zip >>>>>> file) for both the default threshold value >>>>>> and a value of 0.1, and for both 3.6 and 3.10 >>>>>> PETSc versions. >>>>>> >>>>>> For convenience, I summarized the information >>>>>> in a graph, also attached (png file). >>>>>> >>>>>> As you can see, there are slight differences >>>>>> between the two versions but none is >>>>>> critical, in my opinion. Do you see anything >>>>>> suspicious in the outputs? >>>>>> >>>>>> + I can't find the default threshold value. >>>>>> Do you know where I can find it? >>>>>> >>>>>> Thanks for the follow-up >>>>>> >>>>>> Myriam >>>>>> >>>>>> >>>>>> Le 03/05/19 à 14:06, Matthew Knepley a écrit : >>>>>>> On Tue, Mar 5, 2019 at 7:14 AM Myriam >>>>>>> Peyrounette <myriam.peyroune...@idris.fr >>>>>>> <mailto:myriam.peyroune...@idris.fr>> wrote: >>>>>>> >>>>>>> Hi Matt, >>>>>>> >>>>>>> I plotted the memory scalings using >>>>>>> different threshold values. The two >>>>>>> scalings are slightly translated (from >>>>>>> -22 to -88 mB) but this gain is >>>>>>> neglectable. The 3.6-scaling keeps being >>>>>>> robust while the 3.10-scaling deteriorates. >>>>>>> >>>>>>> Do you have any other suggestion? >>>>>>> >>>>>>> Mark, what is the option she can give to >>>>>>> output all the GAMG data? >>>>>>> >>>>>>> Also, run using -ksp_view. GAMG will report >>>>>>> all the sizes of its grids, so it should be >>>>>>> easy to see >>>>>>> if the coarse grid sizes are increasing, and >>>>>>> also what the effect of the threshold value is. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Myriam >>>>>>> >>>>>>> Le 03/02/19 à 02:27, Matthew Knepley a >>>>>>> écrit : >>>>>>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam >>>>>>>> Peyrounette via petsc-users >>>>>>>> <petsc-users@mcs.anl.gov >>>>>>>> <mailto:petsc-users@mcs.anl.gov>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I used to run my code with PETSc >>>>>>>> 3.6. Since I upgraded the PETSc version >>>>>>>> to 3.10, this code has a bad memory >>>>>>>> scaling. >>>>>>>> >>>>>>>> To report this issue, I took the >>>>>>>> PETSc script ex42.c and slightly >>>>>>>> modified it so that the KSP and PC >>>>>>>> configurations are the same as in my >>>>>>>> code. In particular, I use a >>>>>>>> "personnalised" multi-grid method. The >>>>>>>> modifications are indicated by the >>>>>>>> keyword "TopBridge" in the attached >>>>>>>> scripts. >>>>>>>> >>>>>>>> To plot the memory (weak) scaling, >>>>>>>> I ran four calculations for each >>>>>>>> script with increasing problem >>>>>>>> sizes and computations cores: >>>>>>>> >>>>>>>> 1. 100,000 elts on 4 cores >>>>>>>> 2. 1 million elts on 40 cores >>>>>>>> 3. 10 millions elts on 400 cores >>>>>>>> 4. 100 millions elts on 4,000 cores >>>>>>>> >>>>>>>> The resulting graph is also >>>>>>>> attached. The scaling using PETSc 3.10 >>>>>>>> clearly deteriorates for large >>>>>>>> cases, while the one using PETSc 3.6 is >>>>>>>> robust. >>>>>>>> >>>>>>>> After a few tests, I found that the >>>>>>>> scaling is mostly sensitive to the >>>>>>>> use of the AMG method for the >>>>>>>> coarse grid (line 1780 in >>>>>>>> main_ex42_petsc36.cc). In >>>>>>>> particular, the performance strongly >>>>>>>> deteriorates when commenting lines >>>>>>>> 1777 to 1790 (in main_ex42_petsc36.cc). >>>>>>>> >>>>>>>> Do you have any idea of what >>>>>>>> changed between version 3.6 and version >>>>>>>> 3.10 that may imply such degradation? >>>>>>>> >>>>>>>> >>>>>>>> I believe the default values for PCGAMG >>>>>>>> changed between versions. It sounds >>>>>>>> like the coarsening rate >>>>>>>> is not great enough, so that these >>>>>>>> grids are too large. This can be set using: >>>>>>>> >>>>>>>> >>>>>>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >>>>>>>> >>>>>>>> There is some explanation of this >>>>>>>> effect on that page. Let us know if >>>>>>>> setting this does not correct the >>>>>>>> situation. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Let me know if you need further >>>>>>>> information. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> Myriam Peyrounette >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Myriam Peyrounette >>>>>>>> CNRS/IDRIS - HLST >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for >>>>>>>> granted before they begin their >>>>>>>> experiments is infinitely more >>>>>>>> interesting than any results to which >>>>>>>> their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>>>>> >>>>>>> -- >>>>>>> Myriam Peyrounette >>>>>>> CNRS/IDRIS - HLST >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted >>>>>>> before they begin their experiments is >>>>>>> infinitely more interesting than any results >>>>>>> to which their experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/%7Eknepley/> >>>>>> >>>>>> -- >>>>>> Myriam Peyrounette >>>>>> CNRS/IDRIS - HLST >>>>>> -- >>>>>> >>>>> >>>>> -- >>>>> Myriam Peyrounette >>>>> CNRS/IDRIS - HLST >>>>> -- >>>>> >>>> >>>> -- >>>> Myriam Peyrounette >>>> CNRS/IDRIS - HLST >>>> -- >>> >>> -- >>> Myriam Peyrounette >>> CNRS/IDRIS - HLST >>> -- >>> >> >> -- >> Myriam Peyrounette >> CNRS/IDRIS - HLST >> -- >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/%7Eknepley/> > > -- > Myriam Peyrounette > CNRS/IDRIS - HLST > -- > -- Myriam Peyrounette CNRS/IDRIS - HLST --
smime.p7s
Description: Signature cryptographique S/MIME