Hi, I am a bit confused at how to interpret the log_summary results. In my previous log files, I logged everything in that solving staging, including constructing the matrix and vector and the KSPSolve. I then specifically change the code so that each KSPSolve() function is tightly included within the PetscLogStagePush() and PetscLogStagePop() pair so that we exclude the other timings and concentrate on the linear solver. In this way, I still get list of 16 functions in that stage, although I only included one (KSPSolve). They are VecDot VecNorm VecCopy VecSet VecAXPY VecAYPX VecScatterBegin VecScatterEnd MatMult MatSolve MatLUFactorNum KSPSetup KSPSolve PCSetUp PCSetUpOnBlocks PCApply Are these functions called by the KSPSolve() (in this case, I used -ksp_type cg). I suppose the only network communications are done in the function calls VecScatterBegin VecScatterEnd If I am to compute the percentage of communication specifically for KSPSolve(), shall I just use the times of VecScatterBegin & VecScatterEnd devided by the time of KSPSolve? Or shall I use MatMult, like Satish did in his previous emails? I am a bit confused. Please advise.
Thank you very much. Shi --- Satish Balay <balay at mcs.anl.gov> wrote: > > Just looking at 8 proc run [diffusion stage] we > have: > > MatMult : 79 sec > MatMultAdd : 2 sec > VecScatterBegin: 17 sec > VecScatterEnd : 51 sec > > So basically the communication in MatMult/Add is > represented by > VecScatters. Here out of 81 sec total - 68 seconds > are used for > communication [with a load imbalance of 11 for > vecscaterend] > > So - I think MPI performance is reducing scalability > here.. > > Things to try: > > * -vecstatter_rr etc options I sugested earlier > > * install mpich with '--with-device=ch3:ssm' and see > if it makes a difference > > Satish > > --- Event Stage 4: Diffusion > > [x]rhsLtP 297 1.0 1.1017e+02 1.5 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 7 0 0 0 0 39 0 0 > 0 0 0 > [x]rhsGravity 99 1.0 4.2582e+0083.5 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 > 0 0 0 > VecDot 4657 1.0 2.5748e+01 3.2 7.60e+07 > 3.2 0.0e+00 0.0e+00 4.7e+03 1 1 0 0 6 5 3 0 > 0 65 191 > VecNorm 2477 1.0 2.2109e+01 2.2 3.22e+07 > 2.2 0.0e+00 0.0e+00 2.5e+03 1 0 0 0 3 5 2 0 > 0 35 118 > VecScale 594 1.0 2.9330e-02 1.5 2.61e+08 > 1.5 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 1361 > VecCopy 594 1.0 2.7552e-01 1.3 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 0 > VecSet 3665 1.0 6.0793e-01 1.4 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 0 > VecAXPY 5251 1.0 2.5892e+00 1.2 3.31e+08 > 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 4 0 > 0 0 2137 > VecAYPX 1883 1.0 8.6419e-01 1.3 3.62e+08 > 1.3 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 > 0 0 2296 > VecScatterBegin 2873 1.0 1.7569e+01 3.0 0.00e+00 > 0.0 3.8e+04 1.6e+05 0.0e+00 1 0 10 20 0 5 > 0100100 0 0 > VecScatterEnd 2774 1.0 5.1519e+0110.9 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 7 0 0 > 0 0 0 > MatMult 2477 1.0 7.9186e+01 2.4 2.34e+08 > 2.4 3.5e+04 1.7e+05 0.0e+00 3 11 9 20 0 20 48 91 > 98 0 850 > MatMultAdd 297 1.0 2.8161e+00 5.4 4.46e+07 > 2.2 3.6e+03 3.4e+04 0.0e+00 0 0 1 0 0 0 0 9 > 2 0 125 > MatSolve 2477 1.0 6.2245e+01 1.2 1.41e+08 > 1.2 0.0e+00 0.0e+00 0.0e+00 4 10 0 0 0 22 41 0 > 0 0 926 > MatLUFactorNum 3 1.0 2.7686e-01 1.1 2.79e+08 > 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 2016 > MatGetRow 19560420 1.0 5.5195e+01 1.6 > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 > 20 0 0 0 0 0 > KSPSetup 6 1.0 3.0756e-05 2.8 0.00e+00 > 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 0 > KSPSolve 297 1.0 1.3142e+02 1.0 1.31e+08 > 1.1 3.1e+04 1.7e+05 7.1e+03 8 22 8 18 9 50 93 80 > 86100 1001 > PCSetUp 6 1.0 2.7700e-01 1.1 2.78e+08 > 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 2015 > PCSetUpOnBlocks 297 1.0 2.7794e-01 1.1 2.78e+08 > 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 > 0 0 2008 > PCApply 2477 1.0 6.2772e+01 1.2 1.39e+08 > 1.2 0.0e+00 0.0e+00 0.0e+00 4 10 0 0 0 23 41 0 > 0 0 918 > > ____________________________________________________________________________________ It's here! Your new message! Get new email alerts with the free Yahoo! Toolbar. http://tools.search.yahoo.com/toolbar/features/mail/
