> On 24 Mar 2021, at 01:30, Matthew Knepley <[email protected]> wrote:
>
> This is true, but all the PETSc operations are speeding up by a factor 2x. It
> is hard to believe these were run on the same machine.
> For example, VecScale speeds up!?! So it is not network, or optimizations. I
> cannot explain this.
>
VecMDot speeds up by a factor of 8!
Unrelatedly, one thing I see, which _may_ offer potential for much more
speedup, is this:
BuildTwoSided 17548 1.0 4.9331e+00 9.9 0.00e+00 0.0 5.9e+05 4.0e+00 1.8e+04
BuildTwoSidedF 17547 1.0 5.0489e+00 7.3 0.00e+00 0.0 1.2e+06 3.6e+03 1.8e+04
...
MatAssemblyBegin 17547 1.0 8.8252e+00 1.1 0.00e+00 0.0 1.2e+06 3.6e+03 1.8e+04
MatAssemblyEnd 17547 1.0 2.6903e+00 2.8 2.79e+07 2.7 2.1e+02 2.0e+02 1.0e+01
I think these BuildTwoSided calls are coming from the MatAssemblyBegin/End
pairs.
If you preallocate and fill your matrices with zeros in all the possible places
that you might end up putting a non-zero, then calling
MatSetOption(mat, MAT_SUBSET_OFF_PROC_ENTRIES, PETSC_TRUE)
on the matrix you create will reduce this time in BuildTwoSided to almost zero.
Lawrence
