Thanks Lawrence for your suggestion. It did work and the BuildTwoSided time is almost zero now:
BuildTwoSided 4 1.0 1.6279e-0310.1 0.00e+00 0.0 2.0e+02 4.0e+00 4.0e+00 BuildTwoSidedF 3 1.0 1.5969e-0310.5 0.00e+00 0.0 2.0e+02 3.6e+03 3.0e+00 Mohammad On Thu, Mar 25, 2021 at 4:54 AM Lawrence Mitchell <[email protected]> wrote: > > > > On 24 Mar 2021, at 01:30, Matthew Knepley <[email protected]> wrote: > > > > This is true, but all the PETSc operations are speeding up by a factor > 2x. It is hard to believe these were run on the same machine. > > For example, VecScale speeds up!?! So it is not network, or > optimizations. I cannot explain this. > > > > VecMDot speeds up by a factor of 8! > > Unrelatedly, one thing I see, which _may_ offer potential for much more > speedup, is this: > > BuildTwoSided 17548 1.0 4.9331e+00 9.9 0.00e+00 0.0 5.9e+05 4.0e+00 > 1.8e+04 > BuildTwoSidedF 17547 1.0 5.0489e+00 7.3 0.00e+00 0.0 1.2e+06 3.6e+03 > 1.8e+04 > > ... > > MatAssemblyBegin 17547 1.0 8.8252e+00 1.1 0.00e+00 0.0 1.2e+06 3.6e+03 > 1.8e+04 > MatAssemblyEnd 17547 1.0 2.6903e+00 2.8 2.79e+07 2.7 2.1e+02 2.0e+02 > 1.0e+01 > > I think these BuildTwoSided calls are coming from the MatAssemblyBegin/End > pairs. > > If you preallocate and fill your matrices with zeros in all the possible > places that you might end up putting a non-zero, then calling > > MatSetOption(mat, MAT_SUBSET_OFF_PROC_ENTRIES, PETSC_TRUE) > > on the matrix you create will reduce this time in BuildTwoSided to almost > zero. > > Lawrence > >
