On Sat, Jan 22, 2022 at 10:25 AM Jed Brown <j...@jedbrown.org> wrote:
> Mark Adams <mfad...@lbl.gov> writes: > > > On Fri, Jan 21, 2022 at 9:55 PM Barry Smith <bsm...@petsc.dev> wrote: > > > >> > >> Interesting, Is this with all native Kokkos kernels or do some kokkos > >> kernels use rocm? > >> > > > > Ah, good question. I often run with tpl=0 but I did not specify here on > > Crusher. In looking at the log files I see > > > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-crusher/externalpackages/git.kokkos-kernels/src/impl/tpls > > > > Here is a run with tpls turned off. These tpl includes are gone. > > > > It looks pretty much the same. A little slower but that could be noise. > > > > ************************************************************************************************************************ > > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > > > ************************************************************************************************************************ > > We gotta say 160 chars because that's what we use now. > > done as far as streams, does it know to run on the GPU? You don't specify something like -G 1 here for GPUs. I think you just get them all. 11:14 adams/aijkokkos-gpu-logging= crusher:/gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-olcf-crusher streams cc -o MPIVersion.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O3 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-crusher/include -I/opt/rocm-4.5.0/include `pwd`/MPIVersion.c Running streams with '/usr/bin/srun -p batch -N 1 -A csc314_crusher -t 00:10:00 ' using 'NPMAX=128' 1 53355.9207 Rate (MB/s) 2 39565.2208 Rate (MB/s) 0.741534 3 34538.3431 Rate (MB/s) 0.64732 4 32469.3375 Rate (MB/s) 0.608543 5 31041.1569 Rate (MB/s) 0.581776 6 30113.3826 Rate (MB/s) 0.564387 7 29562.5285 Rate (MB/s) 0.554063 8 29228.8090 Rate (MB/s) 0.547808 9 31474.3616 Rate (MB/s) 0.589895 10 31306.7647 Rate (MB/s) 0.586754 11 31147.4674 Rate (MB/s) 0.583768 12 31006.5008 Rate (MB/s) 0.581126 13 30859.4559 Rate (MB/s) 0.57837 14 30796.0587 Rate (MB/s) 0.577182 15 30604.4849 Rate (MB/s) 0.573591 16 30565.4340 Rate (MB/s) 0.572859 17 32421.9349 Rate (MB/s) 0.607654 18 34365.3424 Rate (MB/s) 0.644078 19 36289.4518 Rate (MB/s) 0.680139 20 38194.5300 Rate (MB/s) 0.715845 21 40160.4660 Rate (MB/s) 0.75269 22 42062.3931 Rate (MB/s) 0.788336 23 43890.2036 Rate (MB/s) 0.822593 24 45775.4680 Rate (MB/s) 0.857927 25 47708.8770 Rate (MB/s) 0.894163 26 49559.6810 Rate (MB/s) 0.928851 27 51457.5537 Rate (MB/s) 0.964421 28 53528.3420 Rate (MB/s) 1.00323