Hello,
I'm sorry as I totally messed up here. It turns out that the problem was caused
because there's a previous installation of OpenMPI (v4.1.6) and it was trying
to run the codes compiled against v5 with the mpirun from v4. I always set up
the systems so that the OS picks up the latest MPI version, but it apparently
didn't become effective this time prompting me to the wrong conclusion. I
should have realized of this fact earlier and not waste everyone's time. My
apologies.
Arturo
Gilles Gouaillardet via users wrote:
Hi,
please open an issue on GitHub at https://github.com/open-mpi/ompi/issues
<https://github.com/open-mpi/ompi/issues>
and provide the requested information.
If the compilation failed when configured with --enable-debug, please share the
logs.
the name of the WRF subroutine suggests the crash might occur in
MPI_Comm_split(),
if so, are you able to craft a reproducer that causes the crash?
How many nodes and MPI tasks are needed in order to evidence the crash?
Cheers,
Gilles
On Wed, Jan 31, 2024 at 10:09 PM afernandez via users <users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org> > wrote:
Hello Joseph,
Sorry for the delay but I didn't know if I was missing something yesterday
evening and wanted to double check everything this morning. This is for WRF but
other apps exhibit the same behavior.
* I had no problem with the serial version (and gdb obviously didn't report any
issue).
* I tried compiling with the --enable-debug flag but it was generating errors
during the compilation and never completed.
* I went back to my standard flags for debugging: -g -fbacktrace -ggdb
-fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow. WRF is still
crashing with little extra info vs yesterday:
Backtrace for this error:
#0 0x7f5a4e54451f in ???
at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#1 0x7f5a4e5a73fe in __GI___libc_free
at ./malloc/malloc.c:3368
#2 0x7f5a4c7aa5c3 in ???
#3 0x7f5a4e83b048 in ???
#4 0x7f5a4e7d3ef1 in ???
#5 0x7f5a4e8dab7b in ???
#6 0x8f6bbf in __module_dm_MOD_split_communicator
at /home/ubuntu/WRF-4.5.2/frame/module_dm.f90:5734
#7 0x1879ebd in init_modules_
at /home/ubuntu/WRF-4.5.2/share/init_modules.f90:63
#8 0x406fe4 in __module_wrf_top_MOD_wrf_init
at ../main/module_wrf_top.f90:130
#9 0x405ff3 in wrf
at /home/ubuntu/WRF-4.5.2/main/wrf.f90:22
#10 0x40605c in main
at /home/ubuntu/WRF-4.5.2/main/wrf.f90:6
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node ip-172-31-31-163 exited
on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Any pointers on what might be going on here as this never happened with OMPIv4.
Thanks.
Joseph Schuchart via users wrote:
Hello,
This looks like memory corruption. Do you have more details on what your app is
doing? I don't see any MPI calls inside the call stack. Could you rebuild Open
MPI with debug information enabled (by adding `--enable-debug` to configure)?
If this error occurs on singleton runs (1 process) then you can easily attach
gdb to it to get a better stack trace. Also, valgrind may help pin down the
problem by telling you which memory block is being free'd here.
Thanks
Joseph
On 1/30/24 07:41, afernandez via users wrote:
quote class="gmail_quote" type="cite" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
Hello,
I upgraded one of the systems to v5.0.1 and have compiled everything > exactly
as dozens of previous times with v4. I wasn't expecting any > issue (and the
compilations didn't report anything out of ordinary) > but running several apps
has resulted in error messages such as:
/Backtrace for this error:/
/#0 0x7f7c9571f51f in ???/
/ at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0/
/#1 0x7f7c957823fe in __GI___libc_free/
/ at ./malloc/malloc.c:3368/
/#2 0x7f7c93a635c3 in ???/
/#3 0x7f7c95f84048 in ???/
/#4 0x7f7c95f1cef1 in ???/
/#5 0x7f7c95e34b7b in ???/
/#6 0x6e05be in ???/
/#7 0x6e58d7 in ???/
/#8 0x405d2c in ???/
/#9 0x7f7c95706d8f in __libc_start_call_main/
/ at ../sysdeps/nptl/libc_start_call_main.h:58/
/#10 0x7f7c95706e3f in __libc_start_main_impl/
/ at ../csu/libc-start.c:392/
/#11 0x405d64 in ???/
/#12 0xffffffffffffffff in ???/
OS is Ubuntu 22.04, OpenMPI was built with GCC13.2, and before > building
OpenMPI, I had previously built the hwloc (2.10.0) library at >
/usr/lib/x86_64-linux-gnu. Maybe I'm missing something pretty basic, > but the
problem seems to be related to memory allocation.
Thanks.
