Public bug reported:
================================================================
TITLE (paste into the "Summary" field)
================================================================
singleton MPI_Init() hangs forever with the uct BTL enabled (no InfiniBand
hardware)
================================================================
DESCRIPTION (paste into the bug description field)
================================================================
Any MPI program started directly -- a "singleton", i.e. not launched through
mpirun -- hangs inside MPI_Init() and never returns. The process spins at 100%
CPU on sched_yield(). The same binary run under mpirun works normally, at any
rank count. This machine has no InfiniBand/RDMA hardware.
The hang happens inside MPI_Init(), before the program can print anything, so
it presents as a silent freeze with no diagnostic output at all.
[Impact]
This is not limited to hand-written MPI code. It breaks any MPI-linked
application that is normally run as a single process. I hit it with the OPM
Flow reservoir simulator (flow, flow_comp), which simply froze on startup with
no message; it took a strace to find out why. Users of such applications get a
hang with no indication that MPI is involved.
[Steps to reproduce]
1. sudo apt install libopenmpi-dev
2. Save this as hello.c:
#include <mpi.h>
#include <stdio.h>
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
printf("MPI_Init returned\n");
MPI_Finalize();
return 0;
}
3. mpicc hello.c -o hello
4. ./hello <-- hangs forever, 100% CPU
5. mpirun -np 1 ./hello <-- prints "MPI_Init returned" and exits normally
A self-contained script (reproduce.sh, attached) does all of the above and
prints a verdict. It sets HOME to an empty temporary directory so that only
the packaged defaults are exercised, not any local MCA configuration.
Expected: ./hello prints "MPI_Init returned" and exits.
Actual: ./hello never returns.
[Diagnosis]
strace shows the last activity before the spin is UCX transport setup,
followed by an unbounded sched_yield() loop:
openat(AT_FDCWD, "/etc/libnl/classid", O_RDONLY|O_CLOEXEC) = -1 ENOENT
openat(AT_FDCWD, "/etc/ucx/ucx.conf", O_RDONLY) = -1 ENOENT
mprotect(0x..., 325, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
sched_yield() = 0
sched_yield() = 0
... repeats indefinitely
Excluding the uct BTL is the only setting I found that avoids it:
(packaged defaults) -> hang
OMPI_MCA_btl=^uct -> works
OMPI_MCA_pml=ob1 -> hang
UCX_TLS=tcp,self,sm -> hang
OMPI_MCA_btl_uct_memory_domains=none -> hang
Two of those are worth noting: pml=ob1 alone does not help, so the spin is in
the BTL rather than the PML; and restricting the uct memory domains does not
help either, so it is not merely the device lookup.
[Possible packaging fix]
/etc/openmpi/openmpi-mca-params.conf as shipped already excludes several
transports that misbehave when the corresponding hardware is absent:
btl_base_warn_component_unused=0
mtl = ^ofi
btl = ^openib,ofi
osc = ^pt2pt
but it leaves uct enabled. The uct BTL defaults to
btl_uct_memory_domains = mlx5_0,mlx4_0 (Mellanox devices), which are not
present on this machine. Adding uct to that existing exclusion list, i.e.
btl = ^openib,ofi,uct
would fix the out-of-the-box experience, and follows the precedent already set
by that line. The underlying hang looks like an upstream issue in the uct
BTL's singleton-init path, so it may be worth reporting to Open MPI as well.
[Workaround]
Add to /etc/openmpi/openmpi-mca-params.conf (system-wide) or to
~/.openmpi/mca-params.conf (per user):
btl = ^uct
[System information]
$ lsb_release -rd
Description: Ubuntu 26.04.1 LTS
Release: 26.04
$ uname -srm
Linux 7.0.0-30-generic x86_64
$ apt-cache policy openmpi-bin libucx0
openmpi-bin:
Installed: 5.0.10-1
Candidate: 5.0.10-1
Version table:
*** 5.0.10-1 500
500 http://archive.ubuntu.com/ubuntu resolute/universe amd64 Packages
100 /var/lib/dpkg/status
libucx0:
Installed: 1.20.0+ds-4ubuntu2
Candidate: 1.20.0+ds-4ubuntu2
Version table:
*** 1.20.0+ds-4ubuntu2 500
500 http://archive.ubuntu.com/ubuntu resolute/universe amd64 Packages
100 /var/lib/dpkg/status
No InfiniBand hardware: /sys/class/infiniband does not exist, and lspci lists
no Mellanox or InfiniBand devices.
Caveat: I have only reproduced this on one machine, so the absence of
InfiniBand hardware is a correlation rather than a confirmed trigger.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: openmpi-bin 5.0.10-1
ProcVersionSignature: Ubuntu 7.0.0-30.30-generic 7.0.12
Uname: Linux 7.0.0-30-generic x86_64
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Thu Sep 3 10:49:04 2026
InstallationDate: Installed on 2026-08-31 (3 days ago)
InstallationMedia: Ubuntu 26.04.1 LTS "Resolute Raccoon" - Release amd64
(20260826)
SourcePackage: openmpi
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: openmpi (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug resolute wayland-session
** Attachment added: "script to reproduce the issue"
https://bugs.launchpad.net/bugs/2166319/+attachment/5997159/+files/reproduce.sh
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2166319
Title:
singleton MPI_Init() hangs forever with the uct BTL enabled (no
InfiniBand hardware)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openmpi/+bug/2166319/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
