> On Apr 26, 2022, at 6:40 PM, Matthew Knepley <knep...@gmail.com> wrote: > > On Tue, Apr 26, 2022 at 6:12 PM Justin Chang <jychan...@gmail.com > <mailto:jychan...@gmail.com>> wrote: > I think N/A (not applicable) would be a better message than NaN. Prior to > this mail, even I thought I broke something with these NaN's > > We would need some logic to check for NaN and change the format.
Yes, that should to in another MR. > > Matt > > On Tue, Apr 26, 2022 at 2:49 PM Matthew Knepley <knep...@gmail.com > <mailto:knep...@gmail.com>> wrote: > On Tue, Apr 26, 2022 at 12:03 PM Mark Adams <mfad...@lbl.gov > <mailto:mfad...@lbl.gov>> wrote: > Well, Nans are a clear sign that something is very wrong. > > Barry chose them so that it could not be mistaken for an actual number. > > Matt > > On Tue, Apr 26, 2022 at 11:52 AM Jacob Faibussowitsch <jacob....@gmail.com > <mailto:jacob....@gmail.com>> wrote: > There is an automatic warning that shows when you do run with > `-log_view_gpu_time`, but perhaps there should also be an automatic warning > when *not* running with it. It is unfortunate that NaN is the value printed > as this implies a bug but AFAIK it is unavoidable (Barry can say more on this > though). > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > > On Apr 26, 2022, at 09:48, Jose E. Roman <jro...@dsic.upv.es > > <mailto:jro...@dsic.upv.es>> wrote: > > > > You have to add -log_view_gpu_time > > See https://gitlab.com/petsc/petsc/-/merge_requests/5056 > > <https://gitlab.com/petsc/petsc/-/merge_requests/5056> > > > > Jose > > > > > >> El 26 abr 2022, a las 16:39, Mark Adams <mfad...@lbl.gov > >> <mailto:mfad...@lbl.gov>> escribió: > >> > >> I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing > >> data except the two 'Solve' lines. > >> Just cg/jacobi on snes/ex56. > >> > >> Any ideas? > >> > >> VecTDot 2 1.0 nan nan 1.20e+01 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 100 > >> VecNorm 2 1.0 nan nan 1.00e+01 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 100 > >> VecCopy 2 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 0 > >> VecSet 5 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 0 > >> VecAXPY 4 1.0 nan nan 2.40e+01 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 1 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 100 > >> VecPointwiseMult 1 1.0 nan nan 3.00e+00 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 100 > >> KSPSetUp 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 > >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 > >> 0.00e+00 0 > >> KSPSolve 1 1.0 4.0514e-04 1.0 5.50e+01 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 -nan 0 0.00e+00 0 > >> 0.00e+00 100 > >> SNESSolve 1 1.0 2.2128e-02 1.0 5.55e+05 1.0 0.0e+00 0.0e+00 > >> 0.0e+00 72 56 0 0 0 100100 0 0 0 25 -nan 0 0.00e+00 0 > >> 0.00e+00 0 > > > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
