> I'm not talking about timings, I am talking about seeing where a code dies. >
When codes die within the MPI implementation is basically impossible to provide a backtrace from PETSc, but the MPI implementation may offer some verbose messaging. If this happens quite early in the code (and I presume here we are failing somewhere in the 1-to-all pattern when distributing the plex data structure) running with -log_trace may get you some insight on where this is happening (provided that the offending PETSc routine calls LogEventBegin/End) In my experience, the sad story is that when there are software vendor failures with a large number of processes, I haven't found anything more efficient than a few bisection steps with good old printfs.... > > >> As a side note, one would expect that this kind of behavior is consistent >> across the library. >> But in fact, it isn’t. If you do something like -sub_ksp_view_pmat file >> with, for example, DD preconditioners, you end up with a single file with >> all local matrices scrambled in. >> It would be nice to have the same logic, i.e., file.%p, and be able to >> filter ranks that actually print to file like you suggest. >> These kind of options are less critical than -info so I can just insert >> them in the option database after initialization, if rank == view_this_rank >> PetscOptionsInsertString(), but doing that using command line arguments >> would be easier and more elegant, IMO. >> >> Thanks, >> Pierre >> >> Barry >> >> On Apr 22, 2021, at 11:14 AM, Mark Adams <mfad...@lbl.gov> wrote: >> >> OK, so -info out.%p would do what I asked. Alas, what I really need it to >> reduce the output with 128K processes. >> Thanks, >> >> On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <pie...@joliv.et> wrote: >> >>> >>> >>> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <jacob....@gmail.com> >>> wrote: >>> >>> Not out of the box no. There is -info ::self which makes info only print >>> from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. >>> Best advice would be to print to a file and grep the file for any lines >>> starting with the rank number you want. >>> >>> >>> There is no need for grep, IMHO, just do -info file and then open file.%p >>> >>> Thanks, >>> Pierre >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> >>> On Apr 22, 2021, at 10:16, Mark Adams <mfad...@lbl.gov> wrote: >>> >>> Can I make -info only print from one process? >>> Mark >>> >>> >>> >>> >> >> -- Stefano