Re: [petsc-dev] PETSc amg solver with gpu seems run slowly

2022-03-22 Thread Mark Adams
A few points, but first this is a nice start. If you are interested in working on benchmarking that would be great. If so, read on. * Barry pointed out the SOR issues that are thrashing the memory system. This solve would run faster on the CPU (maybe, 9M eqs is a lot). * Most applications run for

Re: [petsc-dev] PETSc amg solver with gpu seems run slowly

2022-03-22 Thread Barry Smith
Indeed PCSetUp is taking most of the time (79%). In the version of PETSc you are running it is doing a great deal of the setup work on the CPU. You can see there is a lot of data movement between the CPU and GPU (in both directions) during the setup; 64 1.91e+03 54 1.21e+03 90 Clearly, we ne

Re: [petsc-dev] PETSc amg solver with gpu seems run slowly

2022-03-22 Thread Barry Smith
It is using MatSOR 369 1.0 9.1214e+00 1.0 7.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 29 27 0 0 0 29 27 0 0 0 803 0 0 0.00e+00 565 1.35e+03 0 which runs on the CPU not the GPU hence the large amount of time in memory copies and poor performance. We are switchin