Dear Jeff, I did find a way to circumvent this issue for my specific application by spawning less frequently. However, I wanted to at least bring attention to this issue for the OpenMPI community, as it can be reproduced with an alarmingly simple program. Perhaps the user's mailing list is not the ideal place for this. Would you recommend that I report this issue on the developer's mailing list or open a GitHub issue? Best wishes, Thomas Pak
On Mar 16 2019, at 7:40 pm, Jeff Hammond <jeff.scie...@gmail.com> wrote: > Is there perhaps a different way to solve your problem that doesn’t spawn so > much as to hit this issue? > > > I’m not denying there’s an issue here, but in a world of finite human effort > and fallible software, sometimes it’s easiest to just avoid the bugs > altogether. > > Jeff > > On Sat, Mar 16, 2019 at 12:11 PM Thomas Pak <thomas....@maths.ox.ac.uk > (mailto:thomas....@maths.ox.ac.uk)> wrote: > > Dear all, > > > > Does anyone have any clue on what the problem could be here? This seems to > > be a persistent problem present in all currently supported OpenMPI releases > > and indicates that there is a fundamental flaw in how OpenMPI handles > > dynamic process creation. > > > > Best wishes, > > Thomas Pak > > > > > > From: "Thomas Pak" <thomas....@maths.ox.ac.uk > > (mailto:thomas....@maths.ox.ac.uk)> > > To: users@lists.open-mpi.org (mailto:users@lists.open-mpi.org) > > Sent: Friday, 7 December, 2018 17:51:29 > > Subject: [OMPI users] MPI_Comm_spawn leads to pipe leak and other errors > > > > > > > > > > Dear all, > > My MPI application spawns a large number of MPI processes using > > MPI_Comm_spawn over its total lifetime. Unfortunately, I have experienced > > that this results in problems for all currently supported OpenMPI versions > > (2.1, 3.0, 3.1 and 4.0). I have written a short, self-contained program in > > C (included below) that spawns child processes using MPI_Comm_spawn in an > > infinite loop, where each child process exits after writing a message to > > stdout. This short program leads to the following issues: > > In versions 2.1.2 (Ubuntu package) and 2.1.5 (compiled from source), the > > program leads to a pipe leak where pipes keep accumulating over time until > > my MPI application crashes because the maximum number of pipes has been > > reached. > > In versions 3.0.3 and 3.1.3 (both compiled from source), there appears to > > be no pipe leak, but the program crashes with the following error message: > > PMIX_ERROR: UNREACHABLE in file ptl_tcp_component.c at line 1257 > > > > In version 4.0.0 (compiled from source), I have not been able to test this > > issue very thoroughly because mpiexec ignores the --oversubscribe > > command-line flag (as detailed in this GitHub issue > > https://github.com/open-mpi/ompi/issues/6130). This prohibits the > > oversubscription of processor cores, which means that spawning additional > > processes immediately results in an error because "not enough slots" are > > available. A fix for this was proposed recently > > (https://github.com/open-mpi/ompi/pull/6139), but since the v4.0.x > > developer branch is being actively developed right now, I decided not go > > into it. > > I have found one e-mail thread on this mailing list about a similar problem > > (https://www.mail-archive.com/users@lists.open-mpi.org/msg10543.html). In > > this thread, Ralph Castain states that this is a known issue and suggests > > that it is fixed in the then upcoming v1.3.x release. However, version 1.3 > > is no longer supported and the issue has reappeared, hence this did not > > solve the issue. > > I have created a GitHub gist that contains the output from "ompi_info > > --all" of all the OpenMPI installations mentioned here, as well as the > > config.log files for the OpenMPI installations that I compiled from source: > > https://gist.github.com/ThomasPak/1003160e396bb88dff27e53c53121e0c. > > I have also attached the code for the short program that demonstrates these > > issues. For good measure, I have included it directly here as well: > > """ > > #include <stdio.h> > > #include <mpi.h> > > > > int main(int argc, char *argv[]) { > > // Initialize MPI > > MPI_Init(NULL, NULL); > > > > // Get parent > > MPI_Comm parent; > > MPI_Comm_get_parent(&parent); > > > > // If the process was not spawned > > if (parent == MPI_COMM_NULL) { > > > > puts("I was not spawned!"); > > // Spawn child process in loop > > char *cmd = argv[0]; > > char **cmd_argv = MPI_ARGV_NULL; > > int maxprocs = 1; > > MPI_Info info = MPI_INFO_NULL; > > int root = 0; > > MPI_Comm comm = MPI_COMM_SELF; > > MPI_Comm intercomm; > > int *array_of_errcodes = MPI_ERRCODES_IGNORE; > > > > for (;;) { > > MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm, > > &intercomm, array_of_errcodes); > > > > MPI_Comm_disconnect(&intercomm); > > } > > > > // If process was spawned > > } else { > > > > puts("I was spawned!"); > > MPI_Comm_disconnect(&parent); > > } > > > > // Finalize > > MPI_Finalize(); > > > > } > > """ > > > > Thanks in advance and best wishes, > > Thomas Pak > > > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org (mailto:users@lists.open-mpi.org) > > https://lists.open-mpi.org/mailman/listinfo/users > > > > > > > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org (mailto:users@lists.open-mpi.org) > > https://lists.open-mpi.org/mailman/listinfo/users > > > > -- > Jeff Hammond > jeff.scie...@gmail.com (mailto:jeff.scie...@gmail.com) > http://jeffhammond.github.io/ > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users >
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
