On Aug 12, 2013, at 7:19 PM, Barry Smith <bsm...@mcs.anl.gov> wrote:
> > Jed, > > I don't understand your computations. I run > > ./ex29 -ksp_type chebyshev -ksp_max_it 2 -ksp_monitor -pc_type eisenstat > -log_summary -ksp_view -ksp_chebyshev_eigenvalues 1.0,2.0 -ksp_norm_type none > I thought we wanted to use richardson to get PCApplyRichardson_SOR to get invoked and save the MatMults in the KSP, because SOR is a strange PC that can use an existing solution w/o doing residual correction. What am I missing. > > 85 ierr = > MatSOR(eis->A,eis->b[pc->presolvedone-1],eis->omega,(MatSORType)(SOR_ZERO_INITIAL_GUESS > | SOR_LOCAL_FORWARD_SWEEP),0.0,1,1,b);CHKERRQ(ierr); > Breakpoint 2, MatSOR (mat=0x7f91ac1a7e70, b=0x7f91ac1cf470, omega=1, > flag=SOR_EISENSTAT, shift=0, its=1, lits=1, x=0x7f91ac1db070) at matrix.c:3681 > Breakpoint 2, MatSOR (mat=0x7f91ac1a7e70, b=0x7f91ac1d5270, omega=1, > flag=SOR_EISENSTAT, shift=0, its=1, lits=1, > 102 ierr = > MatSOR(eis->A,eis->b[pc->presolvedone],eis->omega,(MatSORType)(SOR_ZERO_INITIAL_GUESS > | SOR_LOCAL_BACKWARD_SWEEP),0.0,1,1,x);CHKERRQ(ierr); > > so get .5 + 1 + 1 + .5 = 3 work units (where a work unit is a matmult). > > I than run snes ex19 with -da_refine 6 so that the matrix ops swamp the > vector ops and get > > MatMult 2 1.0 1.6258e-02 1.0 1.19e+07 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 12 0 0 0 0 12 0 0 0 735 > MatSOR 4 1.0 4.9209e-02 1.0 1.79e+07 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 17 0 0 0 0 17 0 0 0 363 > > note that the MatMult() time and flop counts are included in the SOR time. > > Now I run with > > ./ex29 -ksp_type chebyshev -ksp_max_it 2 -ksp_monitor -pc_type sor > -log_summary -ksp_view -ksp_chebyshev_eigenvalues 1.0,2.0 -ksp_norm_type none > > #1 0x000000010232ca57 in PCApply_SOR (pc=0x7ffb4697a470, x=0x7ffb469dc470, > y=0x7ffb469d0870) at sor.c:35 > 35 ierr = > MatSOR(pc->pmat,x,jac->omega,(MatSORType)flag,jac->fshift,jac->its,jac->lits,y);CHKERRQ(ierr); > (gdb) p flag > $1 = 28 > > Breakpoint 2, MatMult (mat=0x7ffb469a9270, x=0x7ffb469d0870, > y=0x7ffb469dc470) at matrix.c:2212 > Breakpoint 1, MatSOR (mat=0x7ffb469a9270, b=0x7ffb469dc470, omega=1, flag=28, > shift=0, its=1, lits=1, x=0x7ffb469d6670) at > Breakpoint 2, MatMult (mat=0x7ffb469a9270, x=0x7ffb469d6670, > y=0x7ffb469dc470) at matrix.c:2212 > Breakpoint 1, MatSOR (mat=0x7ffb469a9270, b=0x7ffb469dc470, omega=1, flag=28, > shift=0, its=1, lits=1, x=0x7ffb469a1670) at > > so get (.5 + .5) + 1 + (.5 + .5) + 1 + (.5 + .5) = 5 work units > > Note that for the first iteration with a zero initial guess the down AND UP > sweep together can be done in one work unit; the values in the downsweep are > saved and used to save work in the up sweep. > > MatMult 2 1.0 1.1801e-02 1.0 1.16e+07 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 10 0 0 0 0 10 0 0 0 981 > MatSOR 3 1.0 4.6818e-02 1.0 1.78e+07 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 16 0 0 0 0 16 0 0 0 380 > > Thus we see that we save all of the MatMult time which is 2 units of the 5 > units needed with SOR in terms of flops computed so 40% of the work but only > 20% of the time. > > On the post-smooth of the multigrid there is a nonzero initial guess > eisenstat does > > if (nonzero) { > ierr = VecCopy(x,eis->b[pc->presolvedone-1]);CHKERRQ(ierr); > ierr = > MatSOR(eis->A,eis->b[pc->presolvedone-1],eis->omega,SOR_APPLY_UPPER,0.0,1,1,x);CHKERRQ(ierr); > > so an extra .5 work unit > > while Chebychev does the matrix vector product to get the initial residual so > > Eisenstat is 3 units + .5 unit + 1 unit = 4.5 units > SOR 5 units + 1 unit = 6 units > > so for combined pre and post smooth Eisenstat/SOR = 7.5/11 work units > > Sending to petsc-dev so this get archived. > > Barry > > I also realized that the trick of applying the symmetric down then up in only > .5 work units + .5 units by saving the intermediate sums could probably by > used in for future iterations as well (the while (its--) { case in > MatSOR_SeqAIJ()) this would improve the speed of plain old SOR(3) etc a great > deal. At the expensive of even more convoluted code :-). > > Note also the Eisentat introduces several extra flops per grid point in > vector operations so for the 5 point stencil the work units are a poor > measure of actual flops performed. > > On Aug 12, 2013, at 3:13 PM, Jed Brown <jedbr...@mcs.anl.gov> wrote: > >> "Mark F. Adams" <mfad...@lbl.gov> writes: >>> In correction scheme MG (not FAS) pre smoothing has no initial >>> solution but post smoothing does. So if we are doing one symmetric GS >>> (two GS iterations) as smoothers then Eisenstat would save us half the >>> work on the first (of 4) G-S iteration and so about 1/8 reduction in >>> work. Is that correct? >> >> Cheby(2)+SOR with -ksp_norm_type none currently does >> >> (2 units left precond SSOR) + 2*(2 units SSOR + 1 MatMult) = 8 units >> down-smooth >> >> and 9 units for nonzero initial guess (need a MatMult up-front). I >> think Mark is complaining about the half work unit that can be saved in >> the left-precond SSOR starting with zero initial guess. >> >> With Eisenstat, we have >> >> (.5 unit pre-solve with zero guess) + 2*(2 units SSOR) + (1 unit post-solve) >> = 5.5 units down >> >> and 7 units in the up-smoother due to a MatMult and the pre-solve being >> a full sweep. Adding these up, I think I'm seeing a minimum of 16.5 >> work units for SOR pre+post-smoothing as compared to 12.5 work units for >> Eisenstat. >> >> Does this sound right? >