> On Jan 25, 2022, at 11:55 AM, Jed Brown <j...@jedbrown.org> wrote:
>
> Barry Smith <bsm...@petsc.dev> writes:
>
>> Thanks Mark, far more interesting. I've improved the formatting to make it
>> easier to read (and fixed width font for email reading)
>>
>> * Can you do same run with say 10 iterations of Jacobi PC?
>>
>> * PCApply performance (looks like GAMG) is terrible! Problems too small?
>
> This is -pc_type jacobi.
Dang, how come it doesn't warn about all the gamg arguments passed to the
program? I saw them and jump to the wrong conclusion.
How come PCApply is so low while Pointwise mult (which should be all of
PCApply) is high?
>
>> * VecScatter time is completely dominated by SFPack! Junchao what's up with
>> that? Lots of little kernels in the PCApply? PCJACOBI run will help clarify
>> where that is coming from.
>
> It's all in MatMult.
>
> I'd like to see a run that doesn't wait for the GPU.
Indeed
>
>> ------------------------------------------------------------------------------------------------------------------------
>> Event Count Time (sec) Flop
>> --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu
>> - GPU
>> Max Ratio Max Ratio Max Ratio Mess AvgLen
>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count
>> Size %F
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> MatMult 200 1.0 6.7831e-01 1.0 4.91e+10 1.0 1.1e+04 6.6e+04
>> 1.0e+00 9 92 99 79 0 71 92100100 0 579,635 1,014,212 1 2.04e-04
>> 0 0.00e+00 100
>> KSPSolve 1 1.0 9.4550e-01 1.0 5.31e+10 1.0 1.1e+04 6.6e+04
>> 6.0e+02 12100 99 79 94 100100100100100 449,667 893,741 1 2.04e-04
>> 0 0.00e+00 100
>> PCApply 201 1.0 1.6966e-01 1.0 3.09e+08 1.0 0.0e+00 0.0e+00
>> 2.0e+00 2 1 0 0 0 18 1 0 0 0 14,558 16,3941 0 0.00e+00
>> 0 0.00e+00 100
>> VecTDot 401 1.0 5.3642e-02 1.3 1.23e+09 1.0 0.0e+00 0.0e+00
>> 4.0e+02 1 2 0 0 62 5 2 0 0 66 183,716 353,914 0 0.00e+00
>> 0 0.00e+00 100
>> VecNorm 201 1.0 2.2219e-02 1.1 6.17e+08 1.0 0.0e+00 0.0e+00
>> 2.0e+02 0 1 0 0 31 2 1 0 0 33 222,325 303,155 0 0.00e+00
>> 0 0.00e+00 100
>> VecAXPY 400 1.0 2.3017e-02 1.1 1.23e+09 1.0 0.0e+00 0.0e+00
>> 0.0e+00 0 2 0 0 0 2 2 0 0 0 427,091 514,744 0 0.00e+00
>> 0 0.00e+00 100
>> VecAYPX 199 1.0 1.1312e-02 1.1 6.11e+08 1.0 0.0e+00 0.0e+00
>> 0.0e+00 0 1 0 0 0 1 1 0 0 0 432,323 532,889 0 0.00e+00
>> 0 0.00e+00 100
>> VecPointwiseMult 201 1.0 1.0471e-02 1.1 3.09e+08 1.0 0.0e+00 0.0e+00
>> 0.0e+00 0 1 0 0 0 1 1 0 0 0 235,882 290,088 0 0.00e+00
>> 0 0.00e+00 100
>> VecScatterBegin 200 1.0 1.8458e-01 1.1 0.00e+00 0.0 1.1e+04 6.6e+04
>> 1.0e+00 2 0 99 79 0 19 0100100 0 0 0 1 2.04e-04
>> 0 0.00e+00 0
>> VecScatterEnd 200 1.0 1.9007e-02 3.7 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00
>> 0 0.00e+00 0
>> SFPack 200 1.0 1.7309e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00 2 0 0 0 0 18 0 0 0 0 0 0 1 2.04e-04
>> 0 0.00e+00 0
>> SFUnpack 200 1.0 2.3165e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00
>> 0 0.00e+00 0
>>
>>
>>> On Jan 25, 2022, at 8:29 AM, Mark Adams <mfad...@lbl.gov> wrote:
>>>
>>> adding Suyash,
>>>
>>> I found the/a problem. Using ex56, which has a crappy decomposition, using
>>> one MPI process/GPU is much faster than using 8 (64 total). (I am looking
>>> at ex13 to see how much of this is due to the decomposition)
>>> If you only use 8 processes it seems that all 8 are put on the first GPU,
>>> but adding -c8 seems to fix this.
>>> Now the numbers are looking reasonable.
>>>
>>> On Mon, Jan 24, 2022 at 3:24 PM Barry Smith <bsm...@petsc.dev
>>> <mailto:bsm...@petsc.dev>> wrote:
>>>
>>> For this, to start, someone can run
>>>
>>> src/vec/vec/tutorials/performance.c
>>>
>>> and compare the performance to that in the technical report Evaluation of
>>> PETSc on a Heterogeneous Architecture \\ the OLCF Summit System \\ Part I:
>>> Vector Node Performance. Google to find. One does not have to and shouldn't
>>> do an extensive study right now that compares everything, instead one
>>> should run a very small number of different size problems (make them big)
>>> and compare those sizes with what Summit gives. Note you will need to make
>>> sure that performance.c uses the Kokkos backend.
>>>
>>> One hopes for better performance than Summit; if one gets tons worse we
>>> know something is very wrong somewhere. I'd love to see some comparisons.
>>>
>>> Barry
>>>
>>>
>>>> On Jan 24, 2022, at 3:06 PM, Justin Chang <jychan...@gmail.com
>>>> <mailto:jychan...@gmail.com>> wrote:
>>>>
>>>> Also, do you guys have an OLCF liaison? That's actually your better bet if
>>>> you do.
>>>>
>>>> Performance issues with ROCm/Kokkos are pretty common in apps besides just
>>>> PETSc. We have several teams actively working on rectifying this. However,
>>>> I think performance issues can be quicker to identify if we had a more
>>>> "official" and reproducible PETSc GPU benchmark, which I've already
>>>> expressed to some folks in this thread, and as others already commented on
>>>> the difficulty of such a task. Hopefully I will have more time soon to
>>>> illustrate what I am thinking.
>>>>
>>>> On Mon, Jan 24, 2022 at 1:57 PM Justin Chang <jychan...@gmail.com
>>>> <mailto:jychan...@gmail.com>> wrote:
>>>> My name has been called.
>>>>
>>>> Mark, if you're having issues with Crusher, please contact Veronica
>>>> Vergara (vergar...@ornl.gov <mailto:vergar...@ornl.gov>). You can cc me
>>>> (justin.ch...@amd.com <mailto:justin.ch...@amd.com>) in those emails
>>>>
>>>> On Mon, Jan 24, 2022 at 1:49 PM Barry Smith <bsm...@petsc.dev
>>>> <mailto:bsm...@petsc.dev>> wrote:
>>>>
>>>>
>>>>> On Jan 24, 2022, at 2:46 PM, Mark Adams <mfad...@lbl.gov
>>>>> <mailto:mfad...@lbl.gov>> wrote:
>>>>>
>>>>> Yea, CG/Jacobi is as close to a benchmark code as we could want. I could
>>>>> run this on one processor to get cleaner numbers.
>>>>>
>>>>> Is there a designated ECP technical support contact?
>>>>
>>>> Mark, you've forgotten you work for DOE. There isn't a non-ECP technical
>>>> support contact.
>>>>
>>>> But if this is an AMD machine then maybe contact Matt's student Justin
>>>> Chang?
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Mon, Jan 24, 2022 at 2:18 PM Barry Smith <bsm...@petsc.dev
>>>>> <mailto:bsm...@petsc.dev>> wrote:
>>>>>
>>>>> I think you should contact the crusher ECP technical support team and
>>>>> tell them you are getting dismel performance and ask if you should expect
>>>>> better. Don't waste time flogging a dead horse.
>>>>>
>>>>>> On Jan 24, 2022, at 2:16 PM, Matthew Knepley <knep...@gmail.com
>>>>>> <mailto:knep...@gmail.com>> wrote:
>>>>>>
>>>>>> On Mon, Jan 24, 2022 at 2:11 PM Junchao Zhang <junchao.zh...@gmail.com
>>>>>> <mailto:junchao.zh...@gmail.com>> wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 24, 2022 at 12:55 PM Mark Adams <mfad...@lbl.gov
>>>>>> <mailto:mfad...@lbl.gov>> wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 24, 2022 at 1:38 PM Junchao Zhang <junchao.zh...@gmail.com
>>>>>> <mailto:junchao.zh...@gmail.com>> wrote:
>>>>>> Mark, I think you can benchmark individual vector operations, and once
>>>>>> we get reasonable profiling results, we can move to solvers etc.
>>>>>>
>>>>>> Can you suggest a code to run or are you suggesting making a vector
>>>>>> benchmark code?
>>>>>> Make a vector benchmark code, testing vector operations that would be
>>>>>> used in your solver.
>>>>>> Also, we can run MatMult() to see if the profiling result is reasonable.
>>>>>> Only once we get some solid results on basic operations, it is useful to
>>>>>> run big codes.
>>>>>>
>>>>>> So we have to make another throw-away code? Why not just look at the
>>>>>> vector ops in Mark's actual code?
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>
>>>>>> --Junchao Zhang
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 24, 2022 at 12:09 PM Mark Adams <mfad...@lbl.gov
>>>>>> <mailto:mfad...@lbl.gov>> wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 24, 2022 at 12:44 PM Barry Smith <bsm...@petsc.dev
>>>>>> <mailto:bsm...@petsc.dev>> wrote:
>>>>>>
>>>>>> Here except for VecNorm the GPU is used effectively in that most of the
>>>>>> time is time is spent doing real work on the GPU
>>>>>>
>>>>>> VecNorm 402 1.0 4.4100e-01 6.1 1.69e+09 1.0 0.0e+00 0.0e+00
>>>>>> 4.0e+02 0 1 0 0 20 9 1 0 0 33 30230 225393 0 0.00e+00
>>>>>> 0 0.00e+00 100
>>>>>>
>>>>>> Even the dots are very effective, only the VecNorm flop rate over the
>>>>>> full time is much much lower than the vecdot. Which is somehow due to
>>>>>> the use of the GPU or CPU MPI in the allreduce?
>>>>>>
>>>>>> The VecNorm GPU rate is relatively high on Crusher and the CPU rate is
>>>>>> about the same as the other vec ops. I don't know what to make of that.
>>>>>>
>>>>>> But Crusher is clearly not crushing it.
>>>>>>
>>>>>> Junchao: Perhaps we should ask Kokkos if they have any experience with
>>>>>> Crusher that they can share. They could very well find some low level
>>>>>> magic.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jan 24, 2022, at 12:14 PM, Mark Adams <mfad...@lbl.gov
>>>>>>> <mailto:mfad...@lbl.gov>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Mark, can we compare with Spock?
>>>>>>>
>>>>>>> Looks much better. This puts two processes/GPU because there are only 4.
>>>>>>> <jac_out_001_kokkos_Spock_6_1_notpl.txt>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which
>>>>>> their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> <jac_out_001_kokkos_Crusher_159_1.txt>
