Is there perhaps a different way to solve your problem that doesn’t spawn so much as to hit this issue?
I’m not denying there’s an issue here, but in a world of finite human effort and fallible software, sometimes it’s easiest to just avoid the bugs altogether. Jeff On Sat, Mar 16, 2019 at 12:11 PM Thomas Pak <thomas....@maths.ox.ac.uk> wrote: > Dear all, > > Does anyone have any clue on what the problem could be here? This seems to > be a persistent problem present in all currently supported OpenMPI releases > and indicates that there is a fundamental flaw in how OpenMPI handles > dynamic process creation. > > Best wishes, > Thomas Pak > > ------------------------------ > *From: *"Thomas Pak" <thomas....@maths.ox.ac.uk> > *To: *users@lists.open-mpi.org > *Sent: *Friday, 7 December, 2018 17:51:29 > *Subject: *[OMPI users] MPI_Comm_spawn leads to pipe leak and other errors > > Dear all, > > My MPI application spawns a large number of MPI processes using > MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced > that this results in problems for all currently supported OpenMPI versions > (2.1, 3.0, 3.1 and 4.0). I have written a short, self-contained program in > C (included below) that spawns child processes using MPI_Comm_spawn in an > infinite loop, where each child process exits after writing a message to > stdout. This short program leads to the following issues: > > In versions 2.1.2 (Ubuntu package) and 2.1.5 (compiled from source), the > program leads to a pipe leak where pipes keep accumulating over time until > my MPI application crashes because the maximum number of pipes has been > reached. > > In versions 3.0.3 and 3.1.3 (both compiled from source), there appears to > be no pipe leak, but the program crashes with the following error message: > PMIX_ERROR: UNREACHABLE in file ptl_tcp_component.c at line 1257 > > In version 4.0.0 (compiled from source), I have not been able to test this > issue very thoroughly because mpiexec ignores the --oversubscribe > command-line flag (as detailed in this GitHub issue > https://github.com/open-mpi/ompi/issues/6130). This prohibits the > oversubscription of processor cores, which means that spawning additional > processes immediately results in an error because "not enough slots" are > available. A fix for this was proposed recently ( > https://github.com/open-mpi/ompi/pull/6139), but since the v4.0.x > developer branch is being actively developed right now, I decided not go > into it. > > I have found one e-mail thread on this mailing list about a similar > problem ( > https://www.mail-archive.com/users@lists.open-mpi.org/msg10543.html). In > this thread, Ralph Castain states that this is a known issue and suggests > that it is fixed in the then upcoming v1.3.x release. However, version 1.3 > is no longer supported and the issue has reappeared, hence this did not > solve the issue. > > I have created a GitHub gist that contains the output from "ompi_info > --all" of all the OpenMPI installations mentioned here, as well as the > config.log files for the OpenMPI installations that I compiled from source: > https://gist.github.com/ThomasPak/1003160e396bb88dff27e53c53121e0c. > > I have also attached the code for the short program that demonstrates > these issues. For good measure, I have included it directly here as well: > > """ > #include <stdio.h> > #include <mpi.h> > > int main(int argc, char *argv[]) { > > // Initialize MPI > MPI_Init(NULL, NULL); > > // Get parent > MPI_Comm parent; > MPI_Comm_get_parent(&parent); > > // If the process was not spawned > if (parent == MPI_COMM_NULL) { > > puts("I was not spawned!"); > > // Spawn child process in loop > char *cmd = argv[0]; > char **cmd_argv = MPI_ARGV_NULL; > int maxprocs = 1; > MPI_Info info = MPI_INFO_NULL; > int root = 0; > MPI_Comm comm = MPI_COMM_SELF; > MPI_Comm intercomm; > int *array_of_errcodes = MPI_ERRCODES_IGNORE; > > for (;;) { > MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, > &intercomm, array_of_errcodes); > > MPI_Comm_disconnect(&intercomm); > } > > // If process was spawned > } else { > > puts("I was spawned!"); > > MPI_Comm_disconnect(&parent); > } > > // Finalize > MPI_Finalize(); > > } > """ > > Thanks in advance and best wishes, > Thomas Pak > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
