Re: [OMPI users] RUNPATH vs. RPATH
Hi Jeff, > Am 09.08.2022 um 16:17 schrieb Jeff Squyres (jsquyres) via users > : > > Just to follow up on this thread... > > Reuti: I merged the PR on to the main docs branch. They're now live -- we > changed the text: > • here: > https://docs.open-mpi.org/en/main/installing-open-mpi/configure-cli-options/installation.html On this page I read: Using --disable-wrapper-rpath will disable both “runpath” and “rpath” behavior in the wrapper compilers. I would phrase it: Using --disable-wrapper-rpath in addition will disable both “runpath” and “rpath” behavior in the wrapper compilers. (otherwise I get a "configure: error: --enable-wrapper-runpath cannot be selected with --disable-wrapper-rpath") > • and here: > https://docs.open-mpi.org/en/main/installing-open-mpi/configure-cli-options/rpath-and-runpath.html The last command reads `shell$ ./configure LDFLAGS=--enable-new-dtags ...`. But the LDFLAGS will be given to the compiler wrapper, hence it seems to need -Wl,--enable-new-dtags what I used initially to avoid: configure:6591: checking whether the C compiler works configure:6613: gcc --enable-new-dtags conftest.c >&5 cc1: error: unknown pass new-dtags specified in -fenable > Here's the corresponding PR to update the v5.0.x docs: > https://github.com/open-mpi/ompi/pull/10640 > > Specifically, the answer to your original question is twofold: > • It's complicated. 🙂 > • It looks like you did the Right Thing for your environment, but you > might want to check the output of "readelf -d ..." to be sure. > Does that additional text help explain things? Yes, thx a lot for the clarification and update of the documentation. -- Reuti > -- > Jeff Squyres > jsquy...@cisco.com > From: Jeff Squyres (jsquyres) > Sent: Saturday, August 6, 2022 9:36 AM > To: Open MPI Users > Subject: Re: [OMPI users] RUNPATH vs. RPATH > > Reuti -- > > See my disclaimers on other posts about apologies for taking so long to reply! > > This code was written forever ago; I had to dig through it a bit, read the > comments and commit messages, and try to remember why it was done this way. > What I thought would be a 5-minute search turned into a few hours of digging > through code, multiple conversations with Brian, and one pull request (so > far). We don't have a definitive answer yet, but we think we're getting > closer. > > The short version is that what you did appears to be correct: > > ./configure LDFLAGS=-Wl,--enable-new-dtags ... > > The longer answer is that whenever you think you understand the shared > library and run-time linkers, you inevitably find out that you don't. The > complicated cases come from the fact that the handling of rpath and runpath > can be different on different platforms, and there are subtle differences in > their behavior (beyond the initial "search before or after LD_LIBRARY_PATH, > such as the handling of primary and secondary/transitive dependencies). > > The pull request I have so far is https://github.com/open-mpi/ompi/pull/10624 > (rendered here: > https://ompi--10624.org.readthedocs.build/en/10624/installing-open-mpi/configure-cli-options/installation.html). > We're not 100% confident in that text yet, but I think we're close to at > least documenting what the current behavior is. Once we nail that down, we > can talk about whether we want to change that behavior. > > > > From: users on behalf of Reuti via users > > Sent: Friday, July 22, 2022 9:48 AM > To: Open MPI Users > Cc: Reuti; zuelc...@staff.uni-marburg.de > Subject: [OMPI users] RUNPATH vs. RPATH > > Hi, > > using Open MPI 4.1.4 > > $ mpicc --show … > > tells me, that the command line contains "… -Wl,--enable-new-dtags …" so that > even older linkers will include RUNPATH instead of RPATH in the created > dynamic binary. On the other hand, Open MPI itself doesn't use this option > for its own libraries: > > ./liboshmem.so.40.30.2 > ./libmpi_mpifh.so.40.30.0 > ./libmpi.so.40.30.4 > ./libmpi_usempi_ignore_tkr.so.40.30.0 > ./libopen-rte.so.40.30.2 > > Is this intended? > > Setting LD_LIBRARY_PATH will instruct the created binary to look for > libraries first in that location and resolve it, but the loaded library in > turn will then use RPATH inside itself first to load additional libraries. > > (I compile Open MPI in my home directory and move it then to the final > destination for the group; setting OPAL_PREFIX of course. I see a mix of > library locations when I run the created binary on my own with `ldd`.) > > Looks like I can get the intended behavior while configuring Open MPI on this > (older) system: > > $ ./configure … LDFLAGS=-Wl,--enable-new-dtags > > -- Reuti signature.asc Description: Message signed with OpenPGP
[OMPI users] RUNPATH vs. RPATH
Hi, using Open MPI 4.1.4 $ mpicc --show … tells me, that the command line contains "… -Wl,--enable-new-dtags …" so that even older linkers will include RUNPATH instead of RPATH in the created dynamic binary. On the other hand, Open MPI itself doesn't use this option for its own libraries: ./liboshmem.so.40.30.2 ./libmpi_mpifh.so.40.30.0 ./libmpi.so.40.30.4 ./libmpi_usempi_ignore_tkr.so.40.30.0 ./libopen-rte.so.40.30.2 Is this intended? Setting LD_LIBRARY_PATH will instruct the created binary to look for libraries first in that location and resolve it, but the loaded library in turn will then use RPATH inside itself first to load additional libraries. (I compile Open MPI in my home directory and move it then to the final destination for the group; setting OPAL_PREFIX of course. I see a mix of library locations when I run the created binary on my own with `ldd`.) Looks like I can get the intended behavior while configuring Open MPI on this (older) system: $ ./configure … LDFLAGS=-Wl,--enable-new-dtags -- Reuti
Re: [OMPI users] Issues with compilers
Hi, what about putting the "-static-intel" into a configuration file for the Intel compiler. Besides the default configuration, one can have a local one and put the path in an environment variable IFORTCFG (there are other ones for C/C++). $ cat myconf --version $ export IFORTCFG=/home/reuti/myconf $ ifort ifort (IFORT) 19.0.5.281 20190815 -- Reuti > Am 22.01.2021 um 15:49 schrieb Alvaro Payero Pinto via users > : > > Dear Open MPI support team, > > I am trying to install Open MPI with Intel compiler suite for the Fortran > side and GNU compiler suite for the C side. For factors that don’t depend > upon me, I’m not allowed to change the C compiler suite to Intel one since > that would mean an additional license. > > Problem arises with the fact that the installation should not dynamically > depend on Intel libraries, so the flag “-static-intel” (or similar) should be > passed to the Fortran compiler. I’ve seen in the FAQ that this problem is > solved by passing an Autotools option “-Wc,-static-intel” to the variable > LDFLAGS when invoking configure with Intel compilers. This works if both > C/C++ and Fortran compilers are from Intel. However, it crashes if the > compiler suite is mixed since GNU C/C++ does not recognise the > “-static-intel” option. > > Is there any way to bypass this crash and to indicate that such option should > only be passed when using Fortran compiler? > > Configure call to reproduce the crash is made as follows: > > ./configure --prefix=/usr/local/ --libdir=/usr/local/lib64/ > --includedir=/usr/local/include/ CC=gcc CXX=g++ 'FLAGS=-O2 -m64' 'CFLAGS=-O2 > -m64' 'CXXFLAGS=-O2 -m64' FC=ifort 'FCFLAGS=-O2 -m64' > LDFLAGS=-Wc,-static-intel > > Please, find attached the output from configure and config.log. > > Additional data: > > · Operating system SLES12 SP3. > · Open MPI version 4.0.5 > · Intel Fortran compiler version 17.0.6 > · GNU C/C++ compiler version 4.8.5. > > I’ll very much appreciate any help provided to solve this problem. > > Kind regards, > > Álvaro > > >
Re: [OMPI users] segfault in libibverbs.so
Am 27.07.2020 um 21:18 schrieb Prentice Bisbal via users: > Can anyone explain why my job still calls libibverbs when I run it with '-mca > btl ^openib'? A similar behavior I observed too in a mixed cluster where some nodes have InfiniBand and others not. Even checking the node beforehand and applying '-mca btl ^openib' didn't help to suppress the warnings about the missing libibverbs. While in case of IB even more libs are required, at least the libibverbs seems to be required to avoid the warning about its absence in any case (while the job continued despite the warning). [node01:119439] mca_base_component_repository_open: unable to open mca_oob_ud: libibverbs.so.1: cannot open shared object file: No such file or directory (ignored) > If I instead use '-mca btl tcp', my jobs don't segfault. I would assum 'mca > btl ^openib' and '-mca btl tcp' to essentially be equivalent, but there's > obviously a difference in the two. This I didn't check but just ignored the warning later on. Would '-mca btl tcp' also allow local communication without the network being involved and/or replace vader? This was the reason I found '-mca btl ^openib' more appealing than listing all others. -- Reuti > Prentice > > On 7/23/20 3:34 PM, Prentice Bisbal wrote: >> I manage a cluster that is very heterogeneous. Some nodes have InfiniBand, >> while others have 10 Gb/s Ethernet. We recently upgraded to CentOS 7, and >> built a new software stack for CentOS 7. We are using OpenMPI 4.0.3, and we >> are using Slurm 19.05.5 as our job scheduler. >> >> We just noticed that when jobs are sent to the nodes with IB, the segfault >> immediately, with the segfault appearing to come from libibverbs.so. This is >> what I see in the stderr output for one of these failed jobs: >> >> srun: error: greene021: tasks 0-3: Segmentation fault >> >> And here is what I see in the log messages of the compute node where that >> segfault happened: >> >> Jul 23 15:19:41 greene021 kernel: mpihello[7911]: segfault at 7f0635f38910 >> ip 7f0635f49405 sp 7ffe354485a0 error 4 >> Jul 23 15:19:41 greene021 kernel: mpihello[7912]: segfault at 7f23d51ea910 >> ip 7f23d51fb405 sp 7ffef250a9a0 error 4 >> Jul 23 15:19:41 greene021 kernel: in >> libibverbs.so.1.5.22.4[7f23d51ec000+18000] >> Jul 23 15:19:41 greene021 kernel: >> Jul 23 15:19:41 greene021 kernel: mpihello[7909]: segfault at 7ff504ba5910 >> ip 7ff504bb6405 sp 7917ccb0 error 4 >> Jul 23 15:19:41 greene021 kernel: in >> libibverbs.so.1.5.22.4[7ff504ba7000+18000] >> Jul 23 15:19:41 greene021 kernel: >> Jul 23 15:19:41 greene021 kernel: mpihello[7910]: segfault at 7fa58abc5910 >> ip 7fa58abd6405 sp 7ffdde50c0d0 error 4 >> Jul 23 15:19:41 greene021 kernel: in >> libibverbs.so.1.5.22.4[7fa58abc7000+18000] >> Jul 23 15:19:41 greene021 kernel: >> Jul 23 15:19:41 greene021 kernel: in >> libibverbs.so.1.5.22.4[7f0635f3a000+18000] >> Jul 23 15:19:41 greene021 kernel >> >> Any idea what is going on here, or how to debug further? I've been using >> OpenMPI for years, and it usually just works. >> >> I normally start my job with srun like this: >> >> srun ./mpihello >> >> But even if I try to take IB out of the equation by starting the job like >> this: >> >> mpirun -mca btl ^openib ./mpihello >> >> I still get a segfault issue, although the message to stderr is now a little >> different: >> >> -- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code. Per user-direction, the job has been aborted. >> -- >> -- >> mpirun noticed that process rank 1 with PID 8502 on node greene021 exited on >> signal 11 (Segmentation fault). >> -- >> >> The segfaults happens immediately. It seems to happen as soon as MPI_Init() >> is called. The program I'm running is very simple MPI "Hello world!" program. >> >> The output of ompi_info is below my signature, in case that helps. >> >> Prentice >> >> $ ompi_info >> Package: Open MPI u...@host.example.com Distribution >> Open MPI: 4.0.3 >> Open MPI repo revision: v4.0.3 >>Open MPI release date: M
Re: [OMPI users] Moving an installation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Am 24.07.2020 um 18:55 schrieb Lana Deere via users: > I have open-mpi 4.0.4 installed on my desktop and my small test programs are > working. > > I would like to migrate the open-mpi to a cluster and run a larger program > there. When moved, the open-mpi installation is in a different pathname than > it was on my desktop and it doesn't seem to work any longer. I can make the > libraries visible via LD_LIBRARY_PATH but this seems insufficient. Is there > an environment variable which can be used to tell the open-mpi where it is > installed? There is OPAL_PREFIX to be set: https://www.open-mpi.org/faq/?category=building#installdirs - -- Reuti -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAl8bIa0ACgkQo/GbGkBRnRrGywCgj5PHSKdMRwSx3jVB4en+wbmV yG8AniYxICQCHrAsxg/Mbx59YpC9ElvW =y8nX -END PGP SIGNATURE-
Re: [OMPI users] running mpirun with grid
Hi, > Am 06.02.2020 um 21:47 schrieb Kulshrestha, Vipul via users > : > > Hi, > > I need to launch my openmpi application on grid. > > My application is designed to run N processes, where each process would have > M threads. > > To run it without grid, I run it as (say N = 7, M = 2) > % mpirun –np 7 > > The above works well and runs N processes. I am also able to submit it on > grid using below command and it works. > > % qsub –pe orte 7 –l os-redhat6.7* -V –j y –b y –shell no mpirun –np 7 > > > However, the above job allocates only N slots on the grid, when it really is > consuming N*M slots. How do I submit the qsub command so that it reserves the > N*M slots, while starting up N processes? I tried belwo but I get some weird > error from ORTE as pasted below. > > % qsub –pe orte 14 –l os-redhat6.7* -V –j y –b y –shell no mpirun –np 7 > a) You will first have to talk to the admin to provide a fixed allocation rule on all involved nodes, hence e.g. "allocation_rule 2" and name this PE "orte2". Essentially you can be sure to get always 2 slots on each node this way. b) Instead of submitting a binary, you will need a job script where you mangle the provided PE_HOSTFILE to include each node only with a slot count of 1. I.e. Open MPI should think to start only one process per node. You can then use the remaining core for an additional thread. As the original file can't be changed, it has to be copied, adjusted and then PE_HOSTFILE reset to this new file. c) It would be nice, if the admin could prepare already a mangled PE_HOSTFILE (maybe by dividing the slotcount by the last diigit in the PE name) in a parallel prolog and put it in $TMPDIR of the job. As the environemnt variables won't be inherited to the job, you will have to point the environment variable PE_HOSTFILE to the mangled one in your job script in this case too. d) SGE should get the real amount of needed slots of your job during submission, i.e. 14 in your case. This way you will get an allocation of 14 slots, due to the fixed allocation rule "orte2" they are equally distributed. The mangled PE_HOSTFILE will include only one slot per node and Open MPI will start only one process per node for a total of 7. Then you can use OMP_NUM_THREAD=2 or alike to tell your application to start an additional thread per node. The environment variable OMP_NUM_THREAD should also be distributed to the nodes by the option "-x" to `mpirun` (or use MPI itself to distribute this information). Note that in contrast to Torque you get each node only once for sure. AFAIR there was a setting in Torque to allow or disallow mutiple elections of the fixed allocation rule per node. HTH -- Reuti
Re: [OMPI users] Univa Grid Engine and OpenMPI 1.8.7
Hi, Am 12.01.2020 um 07:15 schrieb Lane, William via users: > I'm having problems w/an old openMPI test program which I re-compiled using > OpenMPI 1.8.7 for CentOS 6.3 running Univa Grid Engine 8.6.4. IIRC at that time it was necessary to compile Open MPI explicitly with --with-sge. Any newer version of Open MPI wouldn't work for you, where it detects UGE automatically? > • Are the special PE requirements for Son of Grid Engine needed for > Univa Grid Engine 8.6.4 (in particular qsort_args and/or control_slaves both > being present and set to TRUE) Yes, as "control_slaves" allows `qrsh -inherit …` only when it's set. > • is LD_LIBRARY_PATH required to be set for openMPI 1.8.7 to run > (Univa Grid Engine specifically reports that it has: "removed environment > variable LD_LIBRARY_PATH from submit environment - it is considered a > security issue" when I run my test openMPI 1.8.7 job. I usually don't like `qsub -V …` at all (which I think you are using), as a changed interactive environment might break the job at a later point in time and it's hard to investigate. I prefer setting all necessary environment variable inside the job script itself, so that it is self contained. Maybe they judge it a security issue, as this variable would also be present in case you run a queue prolog/epilog as a different user. For the plain job itself it wouldn't matter IMO. And for any further investigation: which problem do you face in detail? -- Reuti
Re: [OMPI users] mpirun --output-filename behavior
> Am 01.11.2019 um 14:46 schrieb Jeff Squyres (jsquyres) via users > : > > On Nov 1, 2019, at 9:34 AM, Jeff Squyres (jsquyres) via users > wrote: >> >>> Point to make: it would be nice to have an option to suppress the output on >>> stdout and/or stderr when output redirection to file is requested. In my >>> case, having stdout still visible on the terminal is desirable but having a >>> way to suppress output of stderr to the terminal would be immensely helpful. >> >> I do believe that --output-file will write to a *local* file on the node >> where it is running (vs. being sent to mpirun, and mpirun writing to the >> output file). So snipping off the output from being sent to mpirun in the >> first place would actually be an efficiency-gaining feature. > > > Guess what? It turns out that this is another > previously-undocumented-but-already-existing feature. :-) > >mpirun --output-filename foo:nocopy ... > > The ":nocopy" suffix will not emit to stdout/stderr; it will *only* write to > the files. > > You can also comma-delimit / mix this with "nojobid" behavior. For example: > >mpirun --output-filename foo:nocopy,nojobid ... >(ordering of the tokens doesn't matter in the comma-delimited list) > > (I have to admit that I actually LOL'ed when I looked in the code and found > that the feature was already there!) > > For the most part, this whole thing needs to get documented. Especially that the colon is a disallowed character in the directory name. Any suffix :foo will just be removed AFAICS without any error output about foo being an unknown option. -- Reuti > I don't know the timing of when this will happen, but we should probably > also rename this to --output-directory to be a bit more accurate (and > probably keep --output-filename as a deprecated synonym for at least the > duration of the 4.0.x series). > > -- > Jeff Squyres > jsquy...@cisco.com >
Re: [OMPI users] can't run MPI job under SGE
> Am 29.07.2019 um 17:17 schrieb David Laidlaw : > > I will try building a newer ompi version in my home directory, but that will > take me some time. > > qconf is not available to me on any machine. It provides that same error > wherever I am able to try it: > > denied: host "..." is neither submit nor admin host > > Here is what it produces when I have a sysadmin run it: > $ qconf -sconf | egrep "(command|daemon)" > qlogin_command /sysvol/sge.test/bin/qlogin-wrapper > qlogin_daemon/sysvol/sge.test/bin/grid-sshd -i > rlogin_command builtin > rlogin_daemonbuiltin > rsh_command builtin > rsh_daemon builtin That's fine. I wondered whether rsh_* would contain a redirection to `ssh` to (to get the source of the used `ssh` in your error output). -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] TMPDIR for running openMPI job under grid
Hi, Am 26.07.2019 um 21:12 schrieb Kulshrestha, Vipul via users: > Hi, > > I am trying to setup my open-mpi application to run under grid. > > It works sometimes, but sometimes I get the below error. I have contacted my > grid site administrator and the message from them is that they cannot change > the TMPDIR path used in the grid configuration. > > I have tried setting TNPDIR, but it does not help (probably because grid > engine resets it). > > What other alternatives do I have? > > One other curious question is that why does open-mpi creates such a large > name? I understand that part of this path is dependent on TMPDIR value, but > even after that it adds additional unnecessary characters like > “openmpi-sessions-<5 digit number>@_0/ number>”, which could have been shortened to something like “omp-<5 digit > number>@_0/<5 digit number>” and saving 14 characters (almost > 15% of possible length). > > Thanks, > Vipul > > PMIx has detected a temporary directory name that results > in a path that is too long for the Unix domain socket: > > Temp dir: /var/spool/sge/wv2/tmp/<9 digit grid job id>.1.<16 character > queuename>.q/openmpi-sessions-43757@<12character machine name>_0/50671 Personally I find it quite unusual to have the scratch directory being located in /var. Often it's a plain /scratch. Could a symbolic link help? I mean: create it in /tmp and point it to /var/spool/sge/wv2/tmp/<9 digit grid job id>.1.<16 character queuename>.q Then /tmp/$(mktemp -u ) could be shorter which you define as TMPDIR before starting `mpiexec`. === If it happens only occasionally, maybe it depends on the length of the hostname where it's running on? If the admin are nice, the could define a symbolic link directly as /scratch pointing to /var/spool/sge/wv2/tmp and setup in the queue configuration /scratch as being TMPDIR. Effect and location like now, but safes some characters -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] can't run MPI job under SGE
Am 25.07.2019 um 23:00 schrieb David Laidlaw:
> Here is most of the command output when run on a grid machine:
>
> dblade65.dhl(101) mpiexec --version
> mpiexec (OpenRTE) 2.0.2
This is some time old. I would suggest to install a fresh one. You can even
compile one in your home directory and install it e.g. in
$HOME/local/openmpi-3.1.4-gcc_7.4.0-shared ( by --prefix=…intended path…) and
then access this for all your jobs (adjust for your version of gcc). In your
~/.bash_profile and the job script:
DEFAULT_MANPATH="$(manpath -q)"
MY_OMPI="$HOME/local/openmpi-3.1.4_gcc-7.4.0_shared"
export PATH="$MY_OMPI/bin:$PATH"
export LD_LIBRARY_PATH="$MY_OMPI/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export MANPATH="$MY_OMPI/share/man${DEFAULT_MANPATH:+:$DEFAULT_MANPATH}"
unset MY_OMPI
unset DEFAULT_MANPATH
Essentially there is no conflict with the already installed version.
> dblade65.dhl(102) ompi_info | grep grid
> MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component
> v2.0.2)
> dblade65.dhl(103) c
> denied: host "dblade65.cs.brown.edu" is neither submit nor admin host
> dblade65.dhl(104)
On a node it’s ok this way.
> Does that suggest anything?
>
> qconf is restricted to sysadmins, which I am not.
What error is output if you try it anyway? Usually the viewing is always
possible.
> I would note that we are running debian stretch on the cluster machines. On
> some of our other (non-grid) machines, running debian buster, the output is:
>
> cslab3d.dhl(101) mpiexec --version
> mpiexec (OpenRTE) 3.1.3
> Report bugs to http://www.open-mpi.org/community/help/
> cslab3d.dhl(102) ompi_info | grep grid
> MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component
> v3.1.3)
If you compile on such a machine and intend to run it in the cluster it won't
work, as the versions don't match. Therefore the above solution, to use a
personal version available in your $HOME for compiling and running the
applications.
Side note: Open MPI binds the processes to cores by default. In case more than
one MPI job is running on a node one will have to use `mpiexec --bind-to none
…` as otherwise all jobs on this node will use core 0 upwards.
-- Reuti
> Thanks!
>
> -David Laidlaw
>
> On Thu, Jul 25, 2019 at 2:13 PM Reuti wrote:
>
> Am 25.07.2019 um 18:59 schrieb David Laidlaw via users:
>
> > I have been trying to run some MPI jobs under SGE for almost a year without
> > success. What seems like a very simple test program fails; the ingredients
> > of it are below. Any suggestions on any piece of the test, reasons for
> > failure, requests for additional info, configuration thoughts, etc. would
> > be much appreciated. I suspect the linkage between SGIEand MPI, but can't
> > identify the problem. We do have SGE support build into MPI. We also have
> > the SGE parallel environment (PE) set up as described in several places on
> > the web.
> >
> > Many thanks for any input!
>
> Did you compile Open MPI on your own or was it delivered with the Linux
> distribution? That it tries to use `ssh` is quite strange, as nowadays Open
> MPI and others have built-in support to detect that they are running under
> the control of a queuing system. It should use `qrsh` in your case.
>
> What does:
>
> mpiexec --version
> ompi_info | grep grid
>
> reveal? What does:
>
> qconf -sconf | egrep "(command|daemon)"
>
> show?
>
> -- Reuti
>
>
> > Cheers,
> >
> > -David Laidlaw
> >
> >
> >
> >
> > Here is how I submit the job:
> >
> >/usr/bin/qsub /gpfs/main/home/dhl/liggghtsTest/hello2/runme
> >
> >
> > Here is what is in runme:
> >
> > #!/bin/bash
> > #$ -cwd
> > #$ -pe orte_fill 1
> > env PATH="$PATH" /usr/bin/mpirun --mca plm_base_verbose 1 -display-
> > allocation ./hello
> >
> >
> > Here is hello.c:
> >
> > #include
> > #include
> > #include
> > #include
> >
> > int main(int argc, char** argv) {
> > // Initialize the MPI environment
> > MPI_Init(NULL, NULL);
> >
> > // Get the number of processes
> > int world_size;
> > MPI_Comm_size(MPI_COMM_WORLD, &world_size);
> >
> > // Get the rank of the process
> > int world_rank;
> > MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
> >
> > // Get the name of the processor
> > char processor_name[MPI_MAX_PROCESSOR_NAME];
> > int name_len;
> > MPI_Get_processor_name(processor_name, &n
Re: [OMPI users] can't run MPI job under SGE
Am 25.07.2019 um 18:59 schrieb David Laidlaw via users:
> I have been trying to run some MPI jobs under SGE for almost a year without
> success. What seems like a very simple test program fails; the ingredients
> of it are below. Any suggestions on any piece of the test, reasons for
> failure, requests for additional info, configuration thoughts, etc. would be
> much appreciated. I suspect the linkage between SGIEand MPI, but can't
> identify the problem. We do have SGE support build into MPI. We also have
> the SGE parallel environment (PE) set up as described in several places on
> the web.
>
> Many thanks for any input!
Did you compile Open MPI on your own or was it delivered with the Linux
distribution? That it tries to use `ssh` is quite strange, as nowadays Open MPI
and others have built-in support to detect that they are running under the
control of a queuing system. It should use `qrsh` in your case.
What does:
mpiexec --version
ompi_info | grep grid
reveal? What does:
qconf -sconf | egrep "(command|daemon)"
show?
-- Reuti
> Cheers,
>
> -David Laidlaw
>
>
>
>
> Here is how I submit the job:
>
>/usr/bin/qsub /gpfs/main/home/dhl/liggghtsTest/hello2/runme
>
>
> Here is what is in runme:
>
> #!/bin/bash
> #$ -cwd
> #$ -pe orte_fill 1
> env PATH="$PATH" /usr/bin/mpirun --mca plm_base_verbose 1 -display-
> allocation ./hello
>
>
> Here is hello.c:
>
> #include
> #include
> #include
> #include
>
> int main(int argc, char** argv) {
> // Initialize the MPI environment
> MPI_Init(NULL, NULL);
>
> // Get the number of processes
> int world_size;
> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
>
> // Get the rank of the process
> int world_rank;
> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
>
> // Get the name of the processor
> char processor_name[MPI_MAX_PROCESSOR_NAME];
> int name_len;
> MPI_Get_processor_name(processor_name, &name_len);
>
> // Print off a hello world message
> printf("Hello world from processor %s, rank %d out of %d processors\n",
>processor_name, world_rank, world_size);
> // system("printenv");
>
> sleep(15); // sleep for 60 seconds
>
> // Finalize the MPI environment.
> MPI_Finalize();
> }
>
>
> This command will build it:
>
> mpicc hello.c -o hello
>
>
> Running produces the following:
>
> /var/spool/gridengine/execd/dblade01/active_jobs/1895308.1/pe_hostfile
> dblade01.cs.brown.edu 1 shor...@dblade01.cs.brown.edu UNDEFINED
> --
> ORTE was unable to reliably start one or more daemons.
> This usually is caused by:
>
> * not finding the required libraries and/or binaries on
> one or more nodes. Please check your PATH and LD_LIBRARY_PATH
> settings, or configure OMPI with --enable-orterun-prefix-by-default
>
> * lack of authority to execute on one or more specified nodes.
> Please verify your allocation and authorities.
>
> * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
> Please check with your sys admin to determine the correct location to use.
>
> * compilation of the orted with dynamic libraries when static are required
> (e.g., on Cray). Please check your configure cmd line and consider using
> one of the contrib/platform definitions for your system type.
>
> * an inability to create a connection back to mpirun due to a
> lack of common network interfaces and/or no route found between
> them. Please check network connectivity (including firewalls
> and network routing requirements).
> --
>
>
> and:
>
> [dblade01:10902] [[37323,0],0] plm:rsh: final template argv:
> /usr/bin/ssh set path = ( /usr/bin $path ) ; if ( $?
> LD_LIBRARY_PATH == 1 ) set OMPI_have_llp ; if ( $?LD_LIBRARY_PATH
> == 0 ) setenv LD_LIBRARY_PATH /usr/lib ; if ( $?OMPI_have_llp == 1 ) setenv
> LD_LIBRARY_PATH /usr/lib:$LD_LIBRARY_PATH ; if ( $?DYLD_LIBRARY
> _PATH == 1 ) set OMPI_have_dllp ; if ( $?DYLD_LIBRARY_PATH == 0 ) setenv
> DYLD_LIBRARY_PATH /usr/lib ; if ( $?OMPI_have_dllp == 1 ) setenv DY
> LD_LIBRARY_PATH /usr/lib:$DYLD_LIBRARY_PATH ; /usr/bin/orted --hnp-topo-sig
> 0N:2S:0L3:4L2:4L1:4C:4H:x86_64 -mca ess "env" -mca ess_base_jo
> bid "2446000128" -mca ess_base_vpid "" -mca ess_base_num_procs "2" -
> mca orte_hnp_uri "2446000128.0;usock;tcp://10.116.85.90:44791"
> --mca plm_base_verbose "1" -mca plm "rsh" -mca orte_display_alloc "1" -mca
> pmix "^s1,s2,cray"
> ssh_exchange_identification: read: Connection reset by peer
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] MPI_INIT failed 4.0.1
Hi, Am 17.04.2019 um 11:07 schrieb Mahmood Naderan: > Hi, > After successful installation of v4 on a custom location, I see some errors > while the default installation (v2) hasn't. Did you also recompile your application with this version of Open MPI? -- Reuti > $ /share/apps/softwares/openmpi-4.0.1/bin/mpirun --version > mpirun (Open MPI) 4.0.1 > > Report bugs to http://www.open-mpi.org/community/help/ > $ /share/apps/softwares/openmpi-4.0.1/bin/mpirun -np 4 pw.x -i mos2.rlx.in > -- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > ompi_mpi_init: ompi_rte_init failed > --> Returned "(null)" (-43) instead of "Success" (0) > -- > -- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > ompi_mpi_init: ompi_rte_init failed > --> Returned "(null)" (-43) instead of "Success" (0) > -- > *** An error occurred in MPI_Init > *** on a NULL communicator > *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, > ***and potentially your MPI job) > [rocks7.jupiterclusterscu.com:18531] Local abort before MPI_INIT completed > completed successfully, but am not able to aggregate error messages, and not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** on a NULL communicator > *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, > ***and potentially your MPI job) > [rocks7.jupiterclusterscu.com:18532] Local abort before MPI_INIT completed > completed successfully, but am not able to aggregate error messages, and not > able to guarantee that all other processes were killed! > -- > Primary job terminated normally, but 1 process returned > a non-zero exit code. Per user-direction, the job has been aborted. > -- > -- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > ompi_mpi_init: ompi_rte_init failed > --> Returned "(null)" (-43) instead of "Success" (0) > -- > -- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > ompi_mpi_init: ompi_rte_init failed > --> Returned "(null)" (-43) instead of "Success" (0) > -- > *** An error occurred in MPI_Init > *** on a NULL communicator > *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, > ***and potentially your MPI job) > [rocks7.jupiterclusterscu.com:18530] Local abort before MPI_INIT completed > completed successfully, but am not able to aggregate error messages, and not > able to guarantee that all other processes were killed! > *** An error occurred in MPI_Init > *** on a NULL communicator > *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, > ***
Re: [OMPI users] relocating an installation
> Am 10.04.2019 um 14:57 schrieb Dave Love : > > In fact, setting OPAL_PREFIX doesn't work for a relocated tree (with > OMPI 1.10 or 3.0). You also need $OPAL_PREFIX/lib and > $OPAL_PREFIX/lib/openmpi on LD_LIBRARY_PATH (assuming $MPI_LIB=$MPI_HOME/lib): > > $ OPAL_PREFIX=$(pwd)/usr/lib64/openmpi3 ./usr/lib64/openmpi3/bin/mpirun > mpirun true > ./usr/lib64/openmpi3/bin/mpirun: error while loading shared libraries: > libopen-rte.so.40: cannot open shared object file: No such file or directory Does export this OPAL_PREFIX also to all child processes? For me using export OPAL_PREFIX=… beforehand worked up to now. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
> Am 09.04.2019 um 14:52 schrieb Dave Love : > > Reuti writes: > >> export OPAL_PREFIX= >> >> to point it to the new location of installation before you start `mpiexec`. > > Thanks; that's now familiar, and I don't know how I missed it with > strings. > > It should be documented. There is this FAQ entry: https://www.open-mpi.org/faq/?category=building#installdirs -- Reuti > I'd have expected --prefix to have the same > effect, and for there to be an MCA variable. Would there be some > problem with either of those? > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] relocating an installation
Hi, > Am 09.04.2019 um 13:31 schrieb Dave Love : > > Is it possible to use the environment or mpirun flags to run an OMPI > that's been relocated from where it was configured/installed? (Say > you've unpacked a system package that expects to be under /usr and want > to run it from home without containers etc.) I thought that was > possible, but I haven't found a way that works. Using --prefix doesn't > find help files, at least. export OPAL_PREFIX= to point it to the new location of installation before you start `mpiexec`. -- Reuti > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Wrapper Compilers
Hi, Am 25.10.2018 um 20:03 schrieb Jadson Silva: > Hello, > > I would like to know if when we install OpenMPI the wrapper compilers are > also installed. > > When I try to compiler the 'hello' examples I get the following message: > > The program 'mpic++' can be found in the following packages: > * lam4-dev > * libmpich-dev > * libopenmpi-dev > > Should I just install one of those packages? If you compile Open MPI on your own, your get both: the runtime environment (i.e. dynamic libraries and certain commands like `mpiexec`) and the compiler wrappers including the necessary header files. In case it's delivered precompiled by a Linux distribution, they often split the packages to contain the runtime parts in one packages and the development tools in an additional package. If you use these delivered libraries and installed the runtime environment also with the package manager of your distribution, I would suggest to install "libopenmpi-dev" (and only this one to avoid conflicts with wrappers from other MPI implementations). -- Reuti PS: Interesting that they still include LAM/MPI, which was superseded by Open MPI some time ago. ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] MPI advantages over PBS
Hi, Am 31.08.2018 um 13:51 schrieb Diego Avesani: > Dear all, > > thanks a lot. So basically the idea is that MPI is more useful when a lot of > communication occurs in the calibrator. On the other hand, the jobs scheduler > is more appropriate when there are no or few communications. > > Is my final statement correct? In my opinion: no. A job scheduler can serialize the workflow and run one job after the other as free resources provide. Their usage may overlap in certain cases, but MPI and a job scheduler don't compete. -- Reuti > Thanks a lot > > Diego > > > > On Thu, 30 Aug 2018 at 14:55, Patrick Begou > wrote: > Diego, > > what you wont to do is parametric studies. There are specific software > available to do this efficiently (ie reducing the number of runs). Software > can then rely in a job scheduler (PBS, SLURM...) which can launch many > parallel mpi applications at the same time depending on the results of > previous runs. > Look at : > - Dakota https://dakota.sandia.gov/ (open source) > - Modefrontier https://www.esteco.com/modefrontier (commercial) > > Patrick > > Diego Avesani wrote: >> Dear all, >> >> thank you for your answers. I will try to explain better my situation. >> I have written a code and I have parallelized it with openMPI. In particular >> I have a two level palatalization. The first takes care of a parallel code >> program and the second run the parallel code with different input in order >> to get the best solution. In the second level the different runs and output >> have to communicate in order to define the best solution and to modify >> accordingly the input data. These communications have to take place >> different times in the all simulation. >> >> I have read some papers where some people do that with PBS or Microsoft job >> scheduling. >> I opted for openMPI. >> >> What do you think? Can you give me reasons supporting my decision? >> >> Thanks >> >> Diego >> >> >> >> On Sun, 26 Aug 2018 at 00:53, John Hearns via users >> wrote: >> Diego, >> I am sorry but you have different things here. PBS is a resource allocation >> system. It will reserve the use of a compute server, or several compute >> servers, for you to run your parallel job on. PBS can launch the MPI job - >> there are several mechanisms for launching parallel jobs. >> MPI is an API for parallel programming. I would rather say a library, but if >> I'm not wrong MPI is a standard for parallel programming and is technically >> an API. >> >> One piece of advice I would have is that you can run MPI programs from the >> command line. So Google for 'Hello World MPI'. Write your first MPI program >> then use mpirun from the command line. >> >> If you have a cluster which has the PBS batch system you can then use PBS to >> run your MPI program. >> IF that is not clear please let us know what help you need. >> >> >> >> >> >> >> >> >> >> >> >> On Sat, 25 Aug 2018 at 06:54, Diego Avesani wrote: >> Dear all, >> >> I have a philosophical question. >> >> I am reading a lot of papers where people use Portable Batch System or job >> scheduler in order to parallelize their code. >> >> What are the advantages in using MPI instead? >> >> I am writing a report on my code, where of course I use openMPI. So tell me >> please how can I cite you. You deserve all the credits. >> >> Thanks a lot, >> Thanks again, >> >> >> Diego >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> ___ >> users mailing list >> >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > > -- > === > | Equipe M.O.S.T. | | > | Patrick BEGOU | > mailto:patrick.be...@grenoble-inp.fr > | > | LEGI| | > | BP 53 X | Tel 04 76 82 51 35 | > | 38041 GRENOBLE CEDEX| Fax 04 76 82 52 71 | > === > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Are MPI datatypes guaranteed to be compile-time constants?
Hi,
Am 04.09.2018 um 23:22 schrieb Benjamin Brock:
> Are MPI datatypes like MPI_INT and MPI_CHAR guaranteed to be compile-time
> constants? Is this defined by the MPI standard, or in the Open MPI
> implementation?
>
> I've written some template code where MPI datatypes are constexpr members,
> which requires that they be known at compile time. This works in Open MPI
> and MPICH, but not in Spectrum MPI--I'm not sure what they've done that
> breaks this, but I'm trying to file a bug and any information you have about
> whether these are compile-time constants would be useful.
For Open MPI and MPICH you can check the definitions with a command like:
$ find . -type f -exec egrep -H ".*define.*MPI_INT.*" {} \;
and get some constants (both *inside* a compiled directory of each, not the
source directories).
What does Spectrum MPI show here? While Platform-MPI was something unique, I
thought Spectrum MPI is based on Open MPI.
How does this effect manifests in Spectrum MPI? It changes between each
compilation of all your source files, i.e. foo.c sees other values than baz.c,
despite the fact that they work together later on when they are linked?
-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] What happened to orte-submit resp. DVM?
> Am 29.08.2018 um 04:46 schrieb Ralph H Castain : > You must have some stale code because those tools no longer exist. Aha. This code is then by accient still in 3.1.2: $ find openmpi-3.1.2 -name "*submit*" openmpi-3.1.2/orte/orted/orted_submit.h openmpi-3.1.2/orte/orted/orted_submit.c openmpi-3.1.2/orte/orted/.deps/liborted_mpir_la-orted_submit.Plo -- Reuti > Note that we are (gradually) replacing orte-dvm with PRRTE: > > https://github.com/pmix/prrte > > See the “how-to” guides for PRRTE towards the bottom of this page: > https://pmix.org/support/how-to/ > > If you still want to use the orte-dvm tool in OMPI, then you start > applications against it using the “prun” tool - works the same as in PRRTE > > Ralph > > >> On Aug 28, 2018, at 1:38 PM, Reuti wrote: >> >> Hi, >> >> Should orte-submit/ompi-submit still be available in 3.x.y? I can spot the >> source, but it's neither build, nor any man page included. >> >> -- Reuti >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] What happened to orte-submit resp. DVM?
Hi, Should orte-submit/ompi-submit still be available in 3.x.y? I can spot the source, but it's neither build, nor any man page included. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] What happened to orte-submit resp. DVM?
Hi, Should orte-submit/ompi-submit still be available in 3.x.y? I can spot the source, but it's neither build, nor any man page included. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] MPI advantages over PBS
Hi, Am 22.08.2018 um 17:49 schrieb Diego Avesani: > Dear all, > > I have a philosophical question. > > I am reading a lot of papers where people use Portable Batch System or job > scheduler in order to parallelize their code. To parallelize their code? I would more phrase it: "Using a scheduler allows to simultaneously execute compiled applications in a cluster without overloading the nodes; being it serial or parallel applications which are executed several times at the same time or one after the other as available cores permit". Any batch scheduler (like PBS) and any parallel library (like MPI in any implementation [like Open MPI]) don't compete, but cover different aspects of running jobs in a cluster. There are even problems, where you don't gain anything by parallelizing the code: think of the task of rendering 5000 images or apply certain effects to them: if the code is perfectly parallel and cuts the execution time by half with each doubling of the number of cores: the overall time to get the final result of all images stays the same. Essentially in such a situation you can execute several serial instances of an application at the same time in a cluster, which might be referred to "running in parallel" – but depending on the context such a statement might be ambiguous. But if you need the result of the first image resp. computation to decide how to proceed, then it's advantageous to parallelize the application on its own instead. -- Reuti > What are the advantages in using MPI instead? > > I am writing a report on my code, where of course I use openMPI. So tell me > please how can I cite you. You deserve all the credits. > > Thanks a lot, > Thanks again, > > > Diego > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] know which CPU has the maximum value
> Am 10.08.2018 um 17:24 schrieb Diego Avesani : > > Dear all, > I have probably understood. > The trick is to use a real vector and to memorize also the rank. Yes, I thought of this: https://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node79.html -- Reuti > Have I understood correctly? > thanks > > Diego > > > On 10 August 2018 at 17:19, Diego Avesani wrote: > Deal all, > I do not understand how MPI_MINLOC works. it seem locate the maximum in a > vector and not the CPU to which the valur belongs to. > > @ray: and if two has the same value? > > thanks > > > Diego > > > On 10 August 2018 at 17:03, Ray Sheppard wrote: > As a dumb scientist, I would just bcast the value I get back to the group and > ask whoever owns it to kindly reply back with its rank. > Ray > > > On 8/10/2018 10:49 AM, Reuti wrote: > Hi, > > Am 10.08.2018 um 16:39 schrieb Diego Avesani : > > Dear all, > > I have a problem: > In my parallel program each CPU compute a value, let's say eff. > > First of all, I would like to know the maximum value. This for me is quite > simple, > I apply the following: > > CALL MPI_ALLREDUCE(eff, effmaxWorld, 1, MPI_DOUBLE_PRECISION, MPI_MAX, > MPI_MASTER_COMM, MPIworld%iErr) > Would MPI_MAXLOC be sufficient? > > -- Reuti > > > However, I would like also to know to which CPU that value belongs. Is it > possible? > > I have set-up a strange procedure but it works only when all the CPUs has > different values but fails when two of then has the same eff value. > > Is there any intrinsic MPI procedure? > in anternative, > do you have some idea? > > really, really thanks. > Diego > > > Diego > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] know which CPU has the maximum value
Hi, > Am 10.08.2018 um 16:39 schrieb Diego Avesani : > > Dear all, > > I have a problem: > In my parallel program each CPU compute a value, let's say eff. > > First of all, I would like to know the maximum value. This for me is quite > simple, > I apply the following: > > CALL MPI_ALLREDUCE(eff, effmaxWorld, 1, MPI_DOUBLE_PRECISION, MPI_MAX, > MPI_MASTER_COMM, MPIworld%iErr) Would MPI_MAXLOC be sufficient? -- Reuti > However, I would like also to know to which CPU that value belongs. Is it > possible? > > I have set-up a strange procedure but it works only when all the CPUs has > different values but fails when two of then has the same eff value. > > Is there any intrinsic MPI procedure? > in anternative, > do you have some idea? > > really, really thanks. > Diego > > > Diego > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] mpi send/recv pair hangin
> Am 10.04.2018 um 13:37 schrieb Noam Bernstein : > >> On Apr 10, 2018, at 4:20 AM, Reuti wrote: >> >>> >>> Am 10.04.2018 um 01:04 schrieb Noam Bernstein : >>> >>>> On Apr 9, 2018, at 6:36 PM, George Bosilca wrote: >>>> >>>> Noam, >>>> >>>> I have few questions for you. According to your original email you are >>>> using OMPI 3.0.1 (but the hang can also be reproduced with the 3.0.0). >>> >>> Correct. >>> >>>> Also according to your stacktrace I assume it is an x86_64, compiled with >>>> icc. >>> >>> x86_64, yes, but, gcc + ifort. I can test with gcc+gfortran if that’s >>> helpful. >> >> Was there any reason not to choose icc + ifort? > > For historical reasons, we only bought ifort, not the complete compiler > suite. But VASP is 99% fortran, so I doubt it makes a difference in this > case. I see. Sure it's nothing which would change the behavior of VASP, but maybe the interplay with Open MPI compiled with gcc. I try in my compilations to stay with one vendor, being it GCC, PGI or Intel. Looks like icc/icpc is freely available now: https://software.intel.com/en-us/system-studio/choose-download#technical Choosing Linux + Linux as platform to develop and execute seems to be the full icc/icpc incl. the MKL (except the Fortran libs and scaLAPACK – but both are freely available in another package). Only point to take care of, is the location intel/system_studio_2018 where the usual compiler directories are located and not one level above. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] mpi send/recv pair hangin
> Am 10.04.2018 um 01:04 schrieb Noam Bernstein : > >> On Apr 9, 2018, at 6:36 PM, George Bosilca wrote: >> >> Noam, >> >> I have few questions for you. According to your original email you are using >> OMPI 3.0.1 (but the hang can also be reproduced with the 3.0.0). > > Correct. > >> Also according to your stacktrace I assume it is an x86_64, compiled with >> icc. > > x86_64, yes, but, gcc + ifort. I can test with gcc+gfortran if that’s > helpful. Was there any reason not to choose icc + ifort? -- Reuti > >> Is your application multithreaded ? How did you initialized MPI (which level >> of threading) ? Can you send us the opal_config.h file please. > > No, no multithreading, at least not intentionally. I can run with > OMP_NUM_THREADS explicitly 1 if you’d like to exclude that as a possibility. > opal_config.h is attached, from ./opal/include/opal_config.h in the build > directory. > > Noam > > > > > || > |U.S. NAVAL| > |_RESEARCH_| > LABORATORY > > Noam Bernstein, Ph.D. > Center for Materials Physics and Technology > U.S. Naval Research Laboratory > T +1 202 404 8628 F +1 202 404 7546 > https://www.nrl.navy.mil > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] mpi send/recv pair hangin
Hi, > Am 05.04.2018 um 16:16 schrieb Noam Bernstein : > > Hi all - I have a code that uses MPI (vasp), and it’s hanging in a strange > way. Basically, there’s a Cartesian communicator, 4x16 (64 processes total), > and despite the fact that the communication pattern is rather regular, one > particular send/recv pair hangs consistently. Basically, across each row of > 4, task 0 receives from 1,2,3, and tasks 1,2,3 send to 0. On most of the 16 > such sets all those send/recv pairs complete. However, on 2 of them, it > hangs (both the send and recv). I have stack traces (with gdb -p on the > running processes) from what I believe are corresponding send/recv pairs. > > > > This is with OpenMPI 3.0.1 (same for 3.0.0, haven’t checked older versions), > Intel compilers (17.2.174). It seems to be independent of which nodes, always > happens on this pair of calls and happens after the code has been running for > a while, and the same code for the other 14 sets of 4 work fine, suggesting > that it’s an MPI issue, rather than an obvious bug in this code or a hardware > problem. Does anyone have any ideas, either about possible causes or how to > debug things further? Do you use scaLAPACK, and which type of BLAS/LAPACK? I used Intel MKL with the Intel compilers for VASP and found, that using in addition a self-compiled scaLAPACK is working fine in combination with Open MPI. Using Intel scaLAPACK and Intel MPI is also working fine. What I never got working was the combination Intel scaLAPACK and Open MPI – at one point one process got a message from a wrong rank IIRC. I tried both: the Intel supplied Open MPI version of scaLAPACK and also compiling the necessary interface on my own for Open MPI in $MKLROOT/interfaces/mklmpi with identical results. -- Reuti ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] latest Intel CPU bug
Am 04.01.2018 um 23:45 schrieb r...@open-mpi.org: > As more information continues to surface, it is clear that this original > article that spurred this thread was somewhat incomplete - probably released > a little too quickly, before full information was available. There is still > some confusion out there, but the gist from surfing the various articles (and > trimming away the hysteria) appears to be: > > * there are two security issues, both stemming from the same root cause. The > “problem” has actually been around for nearly 20 years, but faster processors > are making it much more visible. > > * one problem (Meltdown) specifically impacts at least Intel, ARM, and AMD > processors. This problem is the one that the kernel patches address as it can > be corrected via software, albeit with some impact that varies based on > application. Those apps that perform lots of kernel services will see larger > impacts than those that don’t use the kernel much. > > * the other problem (Spectre) appears to impact _all_ processors (including, > by some reports, SPARC and Power). This problem lacks a software solution > > * the “problem” is only a problem if you are running on shared nodes - i.e., > if multiple users share a common OS instance as it allows a user to > potentially access the kernel information of the other user. So HPC > installations that allocate complete nodes to a single user might want to > take a closer look before installing the patches. Ditto for your desktop and > laptop - unless someone can gain access to the machine, it isn’t really a > “problem”. Weren't there some PowerPC with strict in-order-execution which could circumvent this? I find a hint about an "EIEIO" command only. Sure, in-order-execution might slow down the system too. -- Reuti > > * containers and VMs don’t fully resolve the problem - the only solution > other than the patches is to limit allocations to single users on a node > > HTH > Ralph > > >> On Jan 3, 2018, at 10:47 AM, r...@open-mpi.org wrote: >> >> Well, it appears from that article that the primary impact comes from >> accessing kernel services. With an OS-bypass network, that shouldn’t happen >> all that frequently, and so I would naively expect the impact to be at the >> lower end of the reported scale for those environments. TCP-based systems, >> though, might be on the other end. >> >> Probably something we’ll only really know after testing. >> >>> On Jan 3, 2018, at 10:24 AM, Noam Bernstein >>> wrote: >>> >>> Out of curiosity, have any of the OpenMPI developers tested (or care to >>> speculate) how strongly affected OpenMPI based codes (just the MPI part, >>> obviously) will be by the proposed Intel CPU memory-mapping-related kernel >>> patches that are all the rage? >>> >>> >>> https://arstechnica.com/gadgets/2018/01/whats-behind-the-intel-design-flaw-forcing-numerous-patches/ >>> >>> >>> Noam >>> ___ >>> users mailing list >>> users@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/users >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] mpifort cannot find libgfortran.so at the correct path
> Am 28.11.2017 um 17:19 schrieb Reuti : > > Hi, > >> Am 28.11.2017 um 15:58 schrieb Vanzo, Davide : >> >> Hello all, >> I am having a very weird problem with mpifort that I cannot understand. >> I am building OpenMPI 1.10.3 with GCC 5.4.0 with EasyBuild and everything >> goes fine until I try to use mpifort to build any MPI Fortran code, which >> fails with the error log you see below. >> The thing I do not understand is why the linker searches for libgfortran.so >> in /usr/lib64 instead of the actual GCC installation path, One additional note on this: often a package contains only a runtime library like: libfoobar.so.1.0 as symbolic link to => libfoobar.so.1.0.5 libfoobar.so.1.0.5 is the real library file A developer package on the other hand contains: libfoobar.a libfoobar.so as symbolic link to either => libfoobar.so.1.0.5 or to => libfoobar.so.1.0 (I see either implementation on a couple of them across the libraries in my openSUSE distribution. Both are woking and it seems to be a taste of the developer of a package which one to use.) During compilation it will look for the target of libfoobar.so up to libfoobar.so.1.0.5. Inside libfoobar.so.1.0.5 it will extract the "-soname" which is libfoobar.so.1.0 and put this name in the application. During execution it will then find libfoobar.so.1.0 in the executable (`readelf -d `) and load the target it points to at runtime. -- Reuti >> where the file is actually located. Both LD_LIBRARY_PATH and LIBRARY_PATH >> contain the correct path to the library. The gfortran compiler works >> correctly for compiling serial code. >> Attached you can also find the ourput of the same command with LD_DEBUG=1. >> Any idea what is going wrong? > > Although 1.10.3 is outdated and the use of a newer Open MPI version > advisable, the culprit seems to lie in: > >> cannot open /usr/lib64/libgfortran.so: No such file or directory > > Does this symbolic link exist? Does it point to your installed one too? > > Maybe the developer package of GCC 5.4.0 is missing. Hence it looks for > libgfortran.so somewhere else and finds only a broken symbolic link. > > -- Reuti > > >> Thank you for your help! >> >> $ ll /accre/arch/easybuild/software/Core/GCCcore/5.4.0/lib64/libgfortran.so >> lrwxrwxrwx 1 buildbot buildbot 20 Nov 27 18:44 >> /accre/arch/easybuild/software/Core/GCCcore/5.4.0/lib64/libgfortran.so -> >> libgfortran.so.3.0.0 >> >> $ mpifort multitask_mpi.f90 >> /accre/arch/easybuild/software/Compiler/GCCcore/5.4.0/binutils/2.26/bin/ld.gold: >> error: cannot open /usr/lib64/libgfortran.so: No such file or directory >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_date_and_time' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_st_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_integer_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_string_trim' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_character_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_integer_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_integer_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_character_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_integer_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_character_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_transfer_character_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_string_trim' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined >> reference to '_gfortran_st_write_done' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: >> undefined reference to '_gfortran_st_write' >> /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: >> undefined refere
Re: [OMPI users] mpifort cannot find libgfortran.so at the correct path
Hi, > Am 28.11.2017 um 15:58 schrieb Vanzo, Davide : > > Hello all, > I am having a very weird problem with mpifort that I cannot understand. > I am building OpenMPI 1.10.3 with GCC 5.4.0 with EasyBuild and everything > goes fine until I try to use mpifort to build any MPI Fortran code, which > fails with the error log you see below. > The thing I do not understand is why the linker searches for libgfortran.so > in /usr/lib64 instead of the actual GCC installation path, where the file is > actually located. Both LD_LIBRARY_PATH and LIBRARY_PATH contain the correct > path to the library. The gfortran compiler works correctly for compiling > serial code. > Attached you can also find the ourput of the same command with LD_DEBUG=1. > Any idea what is going wrong? Although 1.10.3 is outdated and the use of a newer Open MPI version advisable, the culprit seems to lie in: > cannot open /usr/lib64/libgfortran.so: No such file or directory Does this symbolic link exist? Does it point to your installed one too? Maybe the developer package of GCC 5.4.0 is missing. Hence it looks for libgfortran.so somewhere else and finds only a broken symbolic link. -- Reuti > Thank you for your help! > > $ ll /accre/arch/easybuild/software/Core/GCCcore/5.4.0/lib64/libgfortran.so > lrwxrwxrwx 1 buildbot buildbot 20 Nov 27 18:44 > /accre/arch/easybuild/software/Core/GCCcore/5.4.0/lib64/libgfortran.so -> > libgfortran.so.3.0.0 > > $ mpifort multitask_mpi.f90 > /accre/arch/easybuild/software/Compiler/GCCcore/5.4.0/binutils/2.26/bin/ld.gold: > error: cannot open /usr/lib64/libgfortran.so: No such file or directory > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_date_and_time' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_st_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_integer_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_string_trim' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_character_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_integer_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_integer_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_character_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_integer_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_character_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_transfer_character_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_string_trim' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function timestamp_: error: undefined > reference to '_gfortran_st_write_done' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write_done' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write_done' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function p0_receive_output_: error: > undefined reference to '_gfortran_st_write_done' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function MAIN__: error: undefined reference > to '_gfortran_stop_string' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function MAIN__: error: undefined reference > to '_gfortran_transfer_real_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function MAIN__: error: undefined reference > to '_gfortran_stop_string' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function MAIN__: error: undefined reference > to '_gfortran_transfer_real_write' > /tmp/ccpSxqE6.o:multitask_mpi.f90:function MAIN__: error: undefined reference > to '_gfortran_stop_string' > /tmp/ccpSxqE6.o:mu
Re: [OMPI users] Vague error message while executing MPI-Fortran program
Hi, > Am 24.10.2017 um 09:33 schrieb Michael Mauersberger > : > > > > Dear all, > > When compiling and running a Fortran program on Linux (OpenSUSE Leap 42.3) I > get an undefinable error message stating, that some "Boundary Run-Time Check > Failure" ocurred for variable "ARGBLOCK_0.0.2". But this variable I don't > know or use in my code and the compiler is tracing me back to the line of a > "CONTAINS" statement in a module. A `strings * | grep ARGBLOCK` in /opt/intel/compilers_and_libraries_2017.4.196/linux/bin/intel64 reveals: ARGBLOCK_%d ARGBLOCK_REC_%d So it looks like the output is generated on-the-fly and doesn't point to any existing variable. But to which argument of which routine is still unclear. Does the Intel Compile have the feature to output a cross-refernce of all used variables? Maybe it's listed there. -- Reuti > I am using the Intel Fortran Compiler from Intel Composer XE 2013 with the > following Options: > ifort -fPIC -g -traceback -O2 -check all,noarg_temp_created -warn all > > Furthermore, the program uses Intel MKL with the functions > DGETRF, DGETRS, DSYGV, DGEMM, DGGEV > and the C-Library NLopt. > > The complete error message looks like: > > Boundary Run-Time Check Failure for variable 'ARGBLOCK_0.0.2' > > forrtl: error (76): Abort trap signal > Image PCRoutineLineSource > > libc.so.6 7F2BF06CC8D7 Unknown Unknown Unknown > libc.so.6 7F2BF06CDCAA Unknown Unknown Unknown > geops 006A863F Unknown Unknown Unknown > libmodell.so 7F2BF119E54D strukturtest_mod_ 223 > strukturtest_mod.f90 > libmodell.so 7F2BF1184056 modell_start_ 169 > modell_start.f90 > geops 0045D1A3 Unknown Unknown Unknown > geops 0042C2C6 Unknown Unknown Unknown > geops 0040A14C Unknown Unknown Unknown > libc.so.6 7F2BF06B86E5 Unknown Unknown Unknown > geops 0040A049 Unknown Unknown Unknown > === > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 134 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > === > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > > The program has the following structure: > - basic functions linked into static library (*.a), containing only modules > --> using MKL routines > - main program linked into a dynamic library, containing 1 bare subroutine, > modules else > - calling program (executed with mpiexec), calls mentioned subroutine in main > program > > Without the calling program (in Open MPI) the subroutine runs without > problems. But when invoking it with the MPI program I get the error message > above. > > So maybe some of you encountered a similar problem and is able to help me. I > would be really grateful. > > Thanks, > > Michael > > ___ > > Dipl.-Ing. Michael Mauersberger > > Tel. +49 351 463 38099 | Fax +49 351 463 37263 > Marschnerstraße 30, 01307 Dresden > Professur für Luftfahrzeugtechnik | Prof. Dr. Klaus Wolf > > Institut für Luft- und Raumfahrttechnik | Fakultät Maschinenwesen > Technische Universität Dresden > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] Honor host_aliases file for tight SGE integration
Hi, I wonder whether it came ever to the discussion, that SGE can have a similar behavior like Torque/PBS regarding the mangling of hostnames. It's similiar to https://github.com/open-mpi/ompi/issues/2328, in the behavior that a node can have multiple network interfaces and each has an unique name. SGE's operation can be routed to a specific network interface by the use of a file: $SGE_ROOT/$SGE_CELL/common/host_aliases which has the format: Hence in the generated $PE_HOSTFILE the name known to SGE is listed, although the `hostname` command provides the real name. Open MPI would in this case start a `qrsh -inherit …` call instead of forking, as it thinks that these are different machines (assuming an allocation_rule of $PE_SLOTS so that the `mpiexec` is supposed to be on the same machine as the started tasks). I tried to go the "old" way to provide a start_proc_args to the PE to create a symbolic link to `hostname` in $TMPDIR, so that inside the job script an adjusted `hostname` call is available, but obviously Open MPI calls gethostname() directly and not by an external binary. So I mangled the hostname in the created machinefile in the jobscript to feed an "adjusted" $PE_HOSTFILE to Open MPI and then it's working as intended: Open MPI creates forks. Does anyone else need such a patch in Open MPI and is it suitable to be included? -- Reuti PS: Only the headnodes have more than one network interface in our case and hence it's didn't come to my attention up to now, as now there was a need to use also some cores on the headnodes. They are known internally to SGE as "login" and "master", but the external names may be "foo" and "baz" which gethostname() returns. ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Setting LD_LIBRARY_PATH for orted
Am 22.08.2017 um 02:04 schrieb Gilles Gouaillardet:
> Gary,
>
>
> one option (as mentioned in the error message) is to configure Open MPI with
> --enable-orterun-prefix-by-default.
>
> this will force the build process to use rpath, so you do not have to set
> LD_LIBRARY_PATH
>
> this is the easiest option, but cannot be used if you plan to relocate the
> Open MPI installation directory.
There is the tool `chrpath` to change rpath and runpath inside a
binary/library. This has to match relocated directory then.
-- Reuti
> an other option is to use a wrapper for orted.
>
> mpirun --mca orte_launch_agent /.../myorted ...
>
> where myorted is a script that looks like
>
> #!/bin/sh
>
> export LD_LIBRARY_PATH=...
>
> exec /.../bin/orted "$@"
>
>
> you can make this setting system-wide by adding the following line to
> /.../etc/openmpi-mca-params.conf
>
> orte_launch_agent = /.../myorted
>
>
> Cheers,
>
>
> Gilles
>
>
> On 8/22/2017 1:06 AM, Jackson, Gary L. wrote:
>>
>> I’m using a binary distribution of OpenMPI 1.10.2. As linked, it requires
>> certain shared libraries outside of OpenMPI for orted itself to start. So,
>> passing in LD_LIBRARY_PATH with the “-x” flag to mpirun doesn’t do anything:
>>
>> $ mpirun –hostfile ${HOSTFILE} -N 1 -n 2 -x LD_LIBRARY_PATH hostname
>>
>> /path/to/orted: error while loading shared libraries: LIBRARY.so: cannot
>> open shared object file: No such file or directory
>>
>> --
>>
>> ORTE was unable to reliably start one or more daemons.
>>
>> This usually is caused by:
>>
>> * not finding the required libraries and/or binaries on
>>
>> one or more nodes. Please check your PATH and LD_LIBRARY_PATH
>>
>> settings, or configure OMPI with --enable-orterun-prefix-by-default
>>
>> * lack of authority to execute on one or more specified nodes.
>>
>> Please verify your allocation and authorities.
>>
>> * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
>>
>> Please check with your sys admin to determine the correct location to use.
>>
>> * compilation of the orted with dynamic libraries when static are required
>>
>> (e.g., on Cray). Please check your configure cmd line and consider using
>>
>> one of the contrib/platform definitions for your system type.
>>
>> * an inability to create a connection back to mpirun due to a
>>
>> lack of common network interfaces and/or no route found between
>>
>> them. Please check network connectivity (including firewalls
>>
>> and network routing requirements).
>>
>> --
>>
>> How do I get around this cleanly? This works just fine when I set
>> LD_LIBRARY_PATH in my .bashrc, but I’d rather not pollute that if I can
>> avoid it.
>>
>> --
>>
>> Gary Jackson, Ph.D.
>>
>> Johns Hopkins University Applied Physics Laboratory
>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Setting LD_LIBRARY_PATH for orted
Hi,
> Am 21.08.2017 um 18:06 schrieb Jackson, Gary L. :
>
>
> I’m using a binary distribution of OpenMPI 1.10.2. As linked, it requires
> certain shared libraries outside of OpenMPI for orted itself to start. So,
> passing in LD_LIBRARY_PATH with the “-x” flag to mpirun doesn’t do anything:
>
> $ mpirun –hostfile ${HOSTFILE} -N 1 -n 2 -x LD_LIBRARY_PATH hostname
> /path/to/orted: error while loading shared libraries: LIBRARY.so: cannot open
> shared object file: No such file or directory
> --
> ORTE was unable to reliably start one or more daemons.
> This usually is caused by:
>
> * not finding the required libraries and/or binaries on
> one or more nodes. Please check your PATH and LD_LIBRARY_PATH
> settings, or configure OMPI with --enable-orterun-prefix-by-default
>
> * lack of authority to execute on one or more specified nodes.
> Please verify your allocation and authorities.
>
> * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
> Please check with your sys admin to determine the correct location to use.
>
> * compilation of the orted with dynamic libraries when static are required
> (e.g., on Cray). Please check your configure cmd line and consider using
> one of the contrib/platform definitions for your system type.
>
> * an inability to create a connection back to mpirun due to a
> lack of common network interfaces and/or no route found between
> them. Please check network connectivity (including firewalls
> and network routing requirements).
> --
>
> How do I get around this cleanly? This works just fine when I set
> LD_LIBRARY_PATH in my .bashrc, but I’d rather not pollute that if I can avoid
> it.
Do you set or extend the LD_LIBRARY_PATH in your .bashrc?
-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Questions about integration with resource distribution systems
Hi, > Am 01.08.2017 um 18:36 schrieb Dave Love : > > Gilles Gouaillardet writes: > >> Dave, >> >> >> unless you are doing direct launch (for example, use 'srun' instead of >> 'mpirun' under SLURM), >> >> this is the way Open MPI is working : mpirun will use whatever the >> resource manager provides >> >> in order to spawn the remote orted (tm with PBS, qrsh with SGE, srun >> with SLURM, ...). >> >> >> then mpirun/orted will fork&exec the MPI tasks. > > I know quite well how SGE works with openmpi, which isn't special -- > I've done enough work on it. SGE tracks the process tree under orted > just like under bash, even if things daemonize. The OP was correct. > > I should qualify that by noting that ENABLE_ADDGRP_KILL has apparently > never propagated through remote startup, Isn't it a setting inside SGE which the sge_execd is aware of? I never exported any environment variable for this purpose. -- Reuti > so killing those orphans after > VASP crashes may fail, though resource reporting works. (I never > installed a fix for want of a test system, but it's not needed with > Linux cpusets.) > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] Questions about integration with resource distribution systems
> Am 26.07.2017 um 15:09 schrieb r...@open-mpi.org: > > mpirun doesn’t get access to that requirement, nor does it need to do so. SGE > will use the requirement when determining the nodes to allocate. m_mem_free appears to come from Univa GE and is not part of the open source versions. So I can't comment on this for sure, but it seems to set the memory also in cgroups. -- Reuti > mpirun just uses the nodes that SGE provides. > > What your cmd line does is restrict the entire operation on each node (daemon > + 8 procs) to 40GB of memory. OMPI does not support per-process restrictions > other than binding to cpus. > > >> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul >> wrote: >> >> Thanks for a quick response. >> >> I will try building OMPI as suggested. >> >> On the integration with unsupported distribution systems, we cannot use >> script based approach, because often these machines don’t have ssh >> permission in customer environment. I will explore the path of writing orte >> component. At this stage, I don’t understand the effort for the same. >> >> I guess my question 2 was not understood correctly. I used the below command >> as an example for SGE and want to understand the expected behavior for such >> a command. With the below command, I expect to have 8 copies of a.out >> launched with each copy having access to 40GB of memory. Is that correct? I >> am doubtful, because I don’t understand how mpirun gets access to >> information about RAM requirement. >> >> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out >> >> >> Regards, >> Vipul >> >> >> >> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of >> r...@open-mpi.org >> Sent: Tuesday, July 25, 2017 8:16 PM >> To: Open MPI Users >> Subject: Re: [OMPI users] Questions about integration with resource >> distribution systems >> >> >> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul >> wrote: >> >> I have several questions about integration of openmpi with resource queuing >> systems. >> >> 1. >> I understand that openmpi supports integration with various resource >> distribution systems such as SGE, LSF, torque etc. >> >> I need to build an openmpi application that can interact with variety of >> different resource distribution systems, since different customers have >> different systems. Based on my research, it seems that I need to build a >> different openmpi installation to work, e.g. create an installation of >> opempi with grid and create a different installation of openmpi with LSF. Is >> there a way to build a generic installation of openmpi that can be used with >> more than 1 distribution system by using some generic mechanism? >> >> Just to be clear: your application doesn’t depend on the environment in any >> way. Only mpirun does - so if you are distributing an _application_, then >> your question is irrelevant. >> >> If you are distributing OMPI itself, and therefore mpirun, then you can >> build the various components if you first install the headers for that >> environment on your system. It means that you need one machine where all >> those resource managers at least have their headers installed on it. Then >> configure OMPI --with-xxx pointing to each of the RM’s headers so all the >> components get built. When the binary hits your customer’s machine, only >> those components that have active libraries present will execute. >> >> >> 2. >> For integration with LSF/grid, how would I specify the memory (RAM) >> requirement (or some other parameter) to bsub/qsub, when launching mpirun >> command? Will something like below work to ensure that each of the 8 copies >> of a.out have 40 GB memory reserved for them by grid engine? >> >> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out >> >> You’ll have to provide something that is environment dependent, I’m afraid - >> there is no standard out there. >> >> >> >> 3. >> Some of our customers use custom distribution engine (some >> non-industry-standard distribution engine). How can I integrate my openmpi >> application with such system? I would think that it should be possible to do >> that if openmpi launched/managed interaction with the distribution engine >> using some kind of generic mechanism (say, use a configurable command to >> launch, monitor, kill a job and then allow specification of a plugin de
Re: [OMPI users] Questions about integration with resource distribution systems
Hi, > Am 26.07.2017 um 15:03 schrieb Kulshrestha, Vipul > : > > Thanks for a quick response. > > I will try building OMPI as suggested. > > On the integration with unsupported distribution systems, we cannot use > script based approach, because often these machines don’t have ssh permission > in customer environment. I will explore the path of writing orte component. > At this stage, I don’t understand the effort for the same. > > I guess my question 2 was not understood correctly. I used the below command > as an example for SGE and want to understand the expected behavior for such a > command. With the below command, I expect to have 8 copies of a.out launched Yep. > with each copy having access to 40GB of memory. Is that correct? SGE will grant the memory, not Open MPI. This is done by SGE's tight integration and as slave tasks are started by `qrsh -inherit …` and not by a plain `ssh`. This way SGE can keep track of the started processes. > I am doubtful, because I don’t understand how mpirun gets access to > information about RAM requirement. > > qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out In case your application relies on the actual value of "m_mem_free", your application has to request this information. This might different between the various queuing systems though. In SGE one could either use `qstat` to `grep` the information, or (instead of a direct `mpirun`) uses a jobscript which will feed this value in addition to an environment variable, which you can access directly in your application. On a command line it would be: $ qsub –pe orte 8 –b y –v m_mem_free=40G –l m_mem_free=40G –cwd mpirun –np 8 a.out 1. -V might set to many variable. Usually I suggest to forward only environment variables which are necessary for the job. The user could set some environment variable by accident and wonder why the job, which started a couple of days later only, crashes; but submitting exactly the same job again succeeds. 2. The 40G is a string in the environment variable, you may want to use the plain value in bytes there. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Questions about integration with resource distribution systems
Hi, > Am 26.07.2017 um 02:16 schrieb r...@open-mpi.org: > > >> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul >> wrote: >> >> I have several questions about integration of openmpi with resource queuing >> systems. >> >> 1. >> I understand that openmpi supports integration with various resource >> distribution systems such as SGE, LSF, torque etc. >> >> I need to build an openmpi application that can interact with variety of >> different resource distribution systems, since different customers have >> different systems. Based on my research, it seems that I need to build a >> different openmpi installation to work, e.g. create an installation of >> opempi with grid and create a different installation of openmpi with LSF. Is >> there a way to build a generic installation of openmpi that can be used with >> more than 1 distribution system by using some generic mechanism? > > Just to be clear: your application doesn’t depend on the environment in any > way. Only mpirun does - so if you are distributing an _application_, then > your question is irrelevant. > > If you are distributing OMPI itself, and therefore mpirun, then you can build > the various components if you first install the headers for that environment > on your system. It means that you need one machine where all those resource > managers at least have their headers installed on it. Then configure OMPI > --with-xxx pointing to each of the RM’s headers so all the components get > built. When the binary hits your customer’s machine, only those components > that have active libraries present will execute. Just note, that the SGE integration doesn't need any library. It's invoked when Open MPI finds certain environment variables set to sensitive values. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Questions about integration with resource distribution systems
Hi, > Am 26.07.2017 um 00:48 schrieb Kulshrestha, Vipul > : > > I have several questions about integration of openmpi with resource queuing > systems. > > 1. > I understand that openmpi supports integration with various resource > distribution systems such as SGE, LSF, torque etc. > > I need to build an openmpi application that can interact with variety of > different resource distribution systems, since different customers have > different systems. Based on my research, it seems that I need to build a > different openmpi installation to work, e.g. create an installation of opempi > with grid and create a different installation of openmpi with LSF. Is there a > way to build a generic installation of openmpi that can be used with more > than 1 distribution system by using some generic mechanism? > > 2. > For integration with LSF/grid, how would I specify the memory (RAM) > requirement (or some other parameter) to bsub/qsub, when launching mpirun > command? Will something like below work to ensure that each of the 8 copies > of a.out have 40 GB memory reserved for them by grid engine? > > qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out m_mem_free is part of Univa SGE (but not the various free ones of SGE AFAIK). Also: this syntax is for SGE, in LSF it's different. To have this independent from the actual queuing system, one could look into DRMAA V.2 (unfortunately not many are supporting this version of DRMAA). -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No components were able to be opened in the pml framework
Hi, Am 30.05.2017 um 21:51 schrieb may abdelghani: > Hi, > 1) it is /home/may/openmpi/lib > 2) do you source .bashrc also for interactive logins? I do not know > (interactive logins ) ~/.bashrc is only sourced for non-interactive sessions. > do I do ? > echo 'export PATH=/home/may/openmpi/bin/:$PATH' >> ~/.bash_profile > echo 'export INCLUDE=/home/may/openmpi/include/:$INCLUDE ' >> ~/.bash_profile > echo 'export LD_LIBRARY_PATH=/home/may/openmpi/lib/:$LD_LIBRARY_PATH' >> > ~/.bash_profile But don't do this that often. Your files will get longer and longer. IMO it's better to edit the files by hand. And maybe source .bashrc in .bash_profile so that only one place needs to be adjusted all the time. The INCLUDE is not used AFAIK. > NB: I installed openmpi on queuing system This is an additional point: which one? It might be, that you have to put the two exports of PATHS and LD_LIBRARY_PATH in your jobscript instead, in you never want to run the application from the command line in parallel. -- Reuti > > En date de : Mar 30.5.17, Reuti a écrit : > > Objet: Re: [OMPI users] No components were able to be opened in the pml > framework > À: "may abdelghani" > Cc: users@lists.open-mpi.org > Date: Mardi 30 mai 2017, 20h35 > > >> Am > 30.05.2017 um 21:21 schrieb may abdelghani : >> >> Hello, sir >> >> 1. Did you compile > Open MPI on your own? yes >> 2.Did you > move it after the installation? no >> >> I compolled it as follows: >> >> [may@xyz ~]$ mkdir > ~/openmpi >> [may@xyz ~]$ tar xzvf > openmpi-2.0.2.tar.gz >> [may@xyz ~]$ cd > openmpi-2.0.2 >> [may@xyz > openmpi-2.0.2]$ ./configure --prefix=/home/may/openmpi > >> [may@xyz > openmpi-2.0.2]$ make >> [may@xyz > openmpi-2.0.2]$ make install >> >> mpirun envirenemnt variable: >> >> echo 'export > PATH=/home/may/openmpi/bin/:$PATH' >> ~/.bashrc >> echo 'export > INCLUDE=/home/may/openmpi/include/:$INCLUDE ' >> > ~/.bashrc >> echo 'export > LD_LIBRARY_PATH=/home/may/openmpi/lib/:$LD_LIBRARY_PATH' >>> ~/.bashrc > > 1) is it > /home/may/openmpi/lib or /home/may/openmpi/lib64? > 2) do you source .bashrc also for interactive > logins? Otherwise it should go to ~/.bash_profile or > ~/.profile > >> >> > >> En date de : Mar 30.5.17, Reuti > a écrit : >> >> Objet: > Re: [OMPI users] No components were able to be opened in the > pml framework >> À: "may > abdelghani" , > "Open MPI Users" >> Date: Mardi 30 mai 2017, 11h23 >> >> Hi, >> >>> >> Am 30.05.2017 um 10:29 schrieb may > abdelghani via users >> : >>> >>> Hello, >>> >>> What it mean > by the >> following error message: >>> >>> No > components were able to be opened in >> > the pml framework. >>> >>> I have opempi-2.0.2 and I used it > with >> ORCA 4.0.0.2 computational > chemistry program. >> >> Did you compile Open MPI on >> your own? Did you move it after the > installation? >> >> -- > Reuti >> >> > > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No components were able to be opened in the pml framework
> Am 30.05.2017 um 21:21 schrieb may abdelghani : > > Hello, sir > > 1. Did you compile Open MPI on your own? yes > 2.Did you move it after the installation? no > > I compolled it as follows: > > [may@xyz ~]$ mkdir ~/openmpi > [may@xyz ~]$ tar xzvf openmpi-2.0.2.tar.gz > [may@xyz ~]$ cd openmpi-2.0.2 > [may@xyz openmpi-2.0.2]$ ./configure --prefix=/home/may/openmpi > [may@xyz openmpi-2.0.2]$ make > [may@xyz openmpi-2.0.2]$ make install > > mpirun envirenemnt variable: > > echo 'export PATH=/home/may/openmpi/bin/:$PATH' >> ~/.bashrc > echo 'export INCLUDE=/home/may/openmpi/include/:$INCLUDE ' >> ~/.bashrc > echo 'export LD_LIBRARY_PATH=/home/may/openmpi/lib/:$LD_LIBRARY_PATH' >> > ~/.bashrc 1) is it /home/may/openmpi/lib or /home/may/openmpi/lib64? 2) do you source .bashrc also for interactive logins? Otherwise it should go to ~/.bash_profile or ~/.profile > > > En date de : Mar 30.5.17, Reuti a écrit : > > Objet: Re: [OMPI users] No components were able to be opened in the pml > framework > À: "may abdelghani" , "Open MPI Users" > > Date: Mardi 30 mai 2017, 11h23 > > Hi, > >> > Am 30.05.2017 um 10:29 schrieb may abdelghani via users > : >> >> Hello, >> >> What it mean by the > following error message: >> >> No components were able to be opened in > the pml framework. >> >> I have opempi-2.0.2 and I used it with > ORCA 4.0.0.2 computational chemistry program. > > Did you compile Open MPI on > your own? Did you move it after the installation? > > -- Reuti > > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No components were able to be opened in the pml framework
Hi, > Am 30.05.2017 um 10:29 schrieb may abdelghani via users > : > > Hello, > > What it mean by the following error message: > > No components were able to be opened in the pml framework. > > I have opempi-2.0.2 and I used it with ORCA 4.0.0.2 computational chemistry > program. Did you compile Open MPI on your own? Did you move it after the installation? -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OpenMPI installation issue or mpi4py compatibility problem
Hi, Am 23.05.2017 um 05:03 schrieb Tim Jim: > Dear Reuti, > > Thanks for the reply. What options do I have to test whether it has > successfully built? LIke before: can you compile and run mpihello.c this time – all as ordinary user in case you installed the Open MPI into something like $HOME/local/openmpi-2.1.1 and set paths accordingly. There is no need to be root to install a personal Open MPI version in your home directory. -- Reuti > > Thanks and kind regards. > Tim > > On 22 May 2017 at 19:39, Reuti wrote: > Hi, > > > Am 22.05.2017 um 07:22 schrieb Tim Jim : > > > > Hello, > > > > Thanks for your message. I'm trying to get this to work on a single > > machine. > > Ok. > > > > How might you suggest getting OpenMPIworking without python and > > CUDA? > > It looks like it's detected automatically. It should be possible to disable > it with the command line option: > > $ ./configure --without-cuda … > > At the end of the configure step out should liste some lines like: > > Miscellaneous > --- > CUDA support: no > > The mpi4py seems unrelated to the compilation of Open MPI itself AFAICS. > > > > I don't recall setting anything for either, as the only command I had > > run was "./configure --prefix=/opt/openmpi/openmpi-2.1.0" - did it possibly > > pick up the paths by accident? > > > > Regarding the lib directory, I checked that the path physically exists. > > Regarding the final part of the email, is it a problem that 'undefined > > reference' is appearing? > > Yes, it tries to resolve missing symbols and didn't succeed. > > -- Reuti > > > > > > Thanks and regards, > > Tim > > > > On 22 May 2017 at 06:54, Reuti wrote: > > > >> -BEGIN PGP SIGNED MESSAGE- > >> Hash: SHA1 > >> > >> Hi, > >> > >> Am 18.05.2017 um 07:44 schrieb Tim Jim: > >> > >>> Hello, > >>> > >>> I have been having some issues with trying to get OpenMPI working with > >> mpi4py. I've tried to break down my troubleshooting into a few chunks > >> below, and I believe that there are a few, distinct issues that need > >> solving. > >> > >> Are you speaking here of a single machine or a cluster? > >> > >> > >>> Following some troubleshooting in the following link: > >>> https://bitbucket.org/mpi4py/mpi4py/issues/69/building- > >> mpi4py-with-openmpi-gives-error > >>> -the mpi4py folks have suggested it an issue that might be better > >> answered here. > >> > >> First approach would be to get Open MPI working, without CUDA and Python > >> being involved. > >> > >> > >>> In summary, I have attempted to install OpenMPI on Ubuntu 16.04 to the > >> following prefix: /opt/openmpi-openmpi-2.1.0. I have also manually added > >> the following to my .bashrc: > >>> export PATH="/opt/openmpi/openmpi-2.1.0/bin:$PATH" > >>> MPI_DIR=/opt/openmpi/openmpi-2.1.0 > >>> export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH > >> > >> This looks fine, although I don't recall setting MPI_DIR for Open MPI > >> itself. It might be a necessity for mpi4py though. > >> > >> One pitfall might be that "lib" is sometimes being created as "lib64" by > >> `libtool`. I forgot the details when this is happening, but it depends on > >> the version of `libtool` being used. > >> > >> > >>> I later became aware that Ubuntu may handle the LD_LIBRARY_PATH > >> differently > >> > >> I don't think that Ubuntu will do anything different than any other Linux. > >> > >> Did you compile Open MPI on your own, or did you install any repository? > >> > >> Are the CUDA application written by yourself or any freely available > >> applications? > >> > >> - -- Reuti > >> > >> > >>> and instead added a new file containing the library path > >> /opt/openmpi/openmpi-2.1.0/lib to /etc/ld.so.conf.d/openmpi-2-1-0.conf, > >> in the style of everything else in that directory. > >>> > >>> I tried to run "mpicc helloworld.c -o hello.bin" as a test on a demo > >> file (as instructed in the link) to check the installation but I had > >> permission issues, since it was installed win opt. However, when I > >> attempted to run the
Re: [OMPI users] OpenMPI installation issue or mpi4py compatibility problem
Hi, > Am 22.05.2017 um 07:22 schrieb Tim Jim : > > Hello, > > Thanks for your message. I'm trying to get this to work on a single > machine. Ok. > How might you suggest getting OpenMPIworking without python and > CUDA? It looks like it's detected automatically. It should be possible to disable it with the command line option: $ ./configure --without-cuda … At the end of the configure step out should liste some lines like: Miscellaneous --- CUDA support: no The mpi4py seems unrelated to the compilation of Open MPI itself AFAICS. > I don't recall setting anything for either, as the only command I had > run was "./configure --prefix=/opt/openmpi/openmpi-2.1.0" - did it possibly > pick up the paths by accident? > > Regarding the lib directory, I checked that the path physically exists. > Regarding the final part of the email, is it a problem that 'undefined > reference' is appearing? Yes, it tries to resolve missing symbols and didn't succeed. -- Reuti > > Thanks and regards, > Tim > > On 22 May 2017 at 06:54, Reuti wrote: > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Hi, >> >> Am 18.05.2017 um 07:44 schrieb Tim Jim: >> >>> Hello, >>> >>> I have been having some issues with trying to get OpenMPI working with >> mpi4py. I've tried to break down my troubleshooting into a few chunks >> below, and I believe that there are a few, distinct issues that need >> solving. >> >> Are you speaking here of a single machine or a cluster? >> >> >>> Following some troubleshooting in the following link: >>> https://bitbucket.org/mpi4py/mpi4py/issues/69/building- >> mpi4py-with-openmpi-gives-error >>> -the mpi4py folks have suggested it an issue that might be better >> answered here. >> >> First approach would be to get Open MPI working, without CUDA and Python >> being involved. >> >> >>> In summary, I have attempted to install OpenMPI on Ubuntu 16.04 to the >> following prefix: /opt/openmpi-openmpi-2.1.0. I have also manually added >> the following to my .bashrc: >>> export PATH="/opt/openmpi/openmpi-2.1.0/bin:$PATH" >>> MPI_DIR=/opt/openmpi/openmpi-2.1.0 >>> export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH >> >> This looks fine, although I don't recall setting MPI_DIR for Open MPI >> itself. It might be a necessity for mpi4py though. >> >> One pitfall might be that "lib" is sometimes being created as "lib64" by >> `libtool`. I forgot the details when this is happening, but it depends on >> the version of `libtool` being used. >> >> >>> I later became aware that Ubuntu may handle the LD_LIBRARY_PATH >> differently >> >> I don't think that Ubuntu will do anything different than any other Linux. >> >> Did you compile Open MPI on your own, or did you install any repository? >> >> Are the CUDA application written by yourself or any freely available >> applications? >> >> - -- Reuti >> >> >>> and instead added a new file containing the library path >> /opt/openmpi/openmpi-2.1.0/lib to /etc/ld.so.conf.d/openmpi-2-1-0.conf, >> in the style of everything else in that directory. >>> >>> I tried to run "mpicc helloworld.c -o hello.bin" as a test on a demo >> file (as instructed in the link) to check the installation but I had >> permission issues, since it was installed win opt. However, when I >> attempted to run the previous with sudo, or sudo -E, in both cases, mpicc >> could not be found. (Perhaps this is a separate issue with my sudo env) >>> >>> To check that mpicc actually works, I have copied helloworld.c to a >> directory where I could execute mpicc without sudo. On running the above >> command, I receive the following error: >>> >>> mpicc: /usr/local/cuda-8.0/targets/x86_64-linux/lib/libOpenCL.so.1: no >> version information available (required by /opt/openmpi/openmpi-2.1.0/ >> lib/libopen-pal.so.20) >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference >> to `clGetPlatformInfo@OPENCL_1.0' >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference >> to `clGetPlatformIDs@OPENCL_1.0' >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference >> to `clGetDeviceInfo@OPENCL_1.0' >>> /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference &g
Re: [OMPI users] OpenMPI installation issue or mpi4py compatibility problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Am 18.05.2017 um 07:44 schrieb Tim Jim: > Hello, > > I have been having some issues with trying to get OpenMPI working with > mpi4py. I've tried to break down my troubleshooting into a few chunks below, > and I believe that there are a few, distinct issues that need solving. Are you speaking here of a single machine or a cluster? > Following some troubleshooting in the following link: > https://bitbucket.org/mpi4py/mpi4py/issues/69/building-mpi4py-with-openmpi-gives-error > -the mpi4py folks have suggested it an issue that might be better answered > here. First approach would be to get Open MPI working, without CUDA and Python being involved. > In summary, I have attempted to install OpenMPI on Ubuntu 16.04 to the > following prefix: /opt/openmpi-openmpi-2.1.0. I have also manually added the > following to my .bashrc: > export PATH="/opt/openmpi/openmpi-2.1.0/bin:$PATH" > MPI_DIR=/opt/openmpi/openmpi-2.1.0 > export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH This looks fine, although I don't recall setting MPI_DIR for Open MPI itself. It might be a necessity for mpi4py though. One pitfall might be that "lib" is sometimes being created as "lib64" by `libtool`. I forgot the details when this is happening, but it depends on the version of `libtool` being used. > I later became aware that Ubuntu may handle the LD_LIBRARY_PATH differently I don't think that Ubuntu will do anything different than any other Linux. Did you compile Open MPI on your own, or did you install any repository? Are the CUDA application written by yourself or any freely available applications? - -- Reuti > and instead added a new file containing the library path > /opt/openmpi/openmpi-2.1.0/lib to /etc/ld.so.conf.d/openmpi-2-1-0.conf, in > the style of everything else in that directory. > > I tried to run "mpicc helloworld.c -o hello.bin" as a test on a demo file (as > instructed in the link) to check the installation but I had permission > issues, since it was installed win opt. However, when I attempted to run the > previous with sudo, or sudo -E, in both cases, mpicc could not be found. > (Perhaps this is a separate issue with my sudo env) > > To check that mpicc actually works, I have copied helloworld.c to a directory > where I could execute mpicc without sudo. On running the above command, I > receive the following error: > > mpicc: /usr/local/cuda-8.0/targets/x86_64-linux/lib/libOpenCL.so.1: no > version information available (required by > /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20) > /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference to > `clGetPlatformInfo@OPENCL_1.0' > /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference to > `clGetPlatformIDs@OPENCL_1.0' > /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference to > `clGetDeviceInfo@OPENCL_1.0' > /opt/openmpi/openmpi-2.1.0/lib/libopen-pal.so.20: undefined reference to > `clGetDeviceIDs@OPENCL_1.0' > collect2: error: ld returned 1 exit status > > I am unsure if I have an installation or permission issues, and I'd be > grateful if anyone can shed some light based on the trials I've done so far. > (I should add I also have a CUDA installation, which I'd like to leverage > too, if possible). I'm still fairly new to the ins and outs of this, so I may > have missed something obvious. Please let me know if any other info is > required. > > Many thanks and kind regards, > Tim > > -- > > Timothy Jim > PhD Researcher in Aerospace > Creative Flow Research Division, > Institute of Fluid Science, Tohoku University > www.linkedin.com/in/timjim/ > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlkiDKIACgkQo/GbGkBRnRou3wCeNztodNKHRxXBAp5ylz041OMP RIwAoLWkjzK1Qr8eWle/PdZGVUsvdN1M =R6up -END PGP SIGNATURE- ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] MPI the correct solution?
As I think it's not relevant to Open MPI itself, I answered in PM only. -- Reuti > Am 18.05.2017 um 18:55 schrieb do...@mail.com: > > On Tue, 9 May 2017 00:30:38 +0200 > Reuti wrote: >> Hi, >> >> Am 08.05.2017 um 23:25 schrieb David Niklas: >> >>> Hello, >>> I originally ported this question at LQ, but the answer I got back >>> shows rather poor insight on the subject of MPI, so I'm taking the >>> liberty of posting here also. >>> >>> https://www.linuxquestions.org/questions/showthread.php?p=5707962 >>> >>> What I'm trying to do is figure out how/what to use to update an osm >>> file (open street map), in a cross system manner. I know the correct >>> program osmosis and for de/re-compression lbzip2 but how to do this >>> across computers is confusing me, even after a few hours of searching >>> online. >> >> lbzip2 is only thread parallel on a single machine. With pbzip2 you >> mention it's the same, but it exists an MPI version MPIBZIP2 - > I can't find the project, do you have a link? > >> unfortunately it looks unmaintained since 2007. Maybe you can contact >> the author about its state. Without an MPI application like this, the >> MPI library is nothing on its own which would divide and distribute one >> task to several machines automatically. > Well, there might be other ways to cause a program to run on multiple > computers. Perhaps a virtual machine made of of multiple physical > machines? > >> osmosis itself seems to run in serial only (they don't say any word >> whether it uses any parallelism). > Yes, it does run multiple threads, you just start another task (and add a > buffer). I tested this on my machine, I think it is --read-xml > --write-xml and --read-xml-change that start new threads. The question is > whether or not java is naively MPI aware or does the app need special > coding? > >> For the intended task the only option is to use a single machine with >> as many cores as possible AFAICS. > Though about that, and it is doable with respect to memory and disk > constraints, the problem is that it would take a *long* time esp. with the > amount of updates I must do, hence my inquiry. > > Thanks, > David > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] IBM Spectrum MPI problem
Hi, > Am 18.05.2017 um 14:02 schrieb Gabriele Fatigati : > > Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0 I noticed this on IBM's website too. Is this freely available? Up to now I was always bounced back to their former Platform MPI when trying to download the community edition (even the evaluation link on the Spectrum MPI page does the same). -- Reuti > based on OpenMPI, so I hope there are some MPI expert can help me to solve > the problem. > > When I run a simple Hello World MPI program, I get the follow error message: > > A requested component was not found, or was unable to be opened. This > means that this component is either not installed or is unable to be > used on your system (e.g., sometimes this means that shared libraries > that the component requires are unable to be found/loaded). Note that > Open MPI stopped checking at the first component that it did not find. > > Host: openpower > Framework: pml > Component: pami > -- > -- > It looks like MPI_INIT failed for some reason; your parallel process is > likely to abort. There are many reasons that a parallel process can > fail during MPI_INIT; some of which are due to configuration or environment > problems. This failure appears to be an internal failure; here's some > additional information (which may only be relevant to an Open MPI > developer): > > mca_pml_base_open() failed > --> Returned "Not found" (-13) instead of "Success" (0) > -- > *** An error occurred in MPI_Init > *** on a NULL communicator > *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, > ***and potentially your MPI job) > > My sysadmin used official IBM Spectrum packages to install MPI, so It's quite > strange that there are some components missing (pami). Any help? Thanks > > -- > Ing. Gabriele Fatigati > > HPC specialist > > SuperComputing Applications and Innovation Department > > Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy > > www.cineca.itTel: +39 051 6171722 > > g.fatigati [AT] cineca.it > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] MPI the correct solution?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Am 08.05.2017 um 23:25 schrieb David Niklas: > Hello, > I originally ported this question at LQ, but the answer I got back shows > rather poor insight on the subject of MPI, so I'm taking the liberty of > posting here also. > > https://www.linuxquestions.org/questions/showthread.php?p=5707962 > > What I'm trying to do is figure out how/what to use to update an osm file > (open street map), in a cross system manner. I know the correct program > osmosis and for de/re-compression lbzip2 but how to do this across > computers is confusing me, even after a few hours of searching online. lbzip2 is only thread parallel on a single machine. With pbzip2 you mention it's the same, but it exists an MPI version MPIBZIP2 - unfortunately it looks unmaintained since 2007. Maybe you can contact the author about its state. Without an MPI application like this, the MPI library is nothing on its own which would divide and distribute one task to several machines automatically. osmosis itself seems to run in serial only (they don't say any word whether it uses any parallelism). For the intended task the only option is to use a single machine with as many cores as possible AFAICS. - -- Reuti -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlkQ8Y8ACgkQo/GbGkBRnRq4jgCeKI39e2U22qsx9f6VeNZyUqNK QzQAoNsbE7yS1SuQpabW67z+7oTGQ7QP =uSOG -END PGP SIGNATURE- ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Behavior of `ompi_info`
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am 25.04.2017 um 17:27 schrieb Reuti: > Hi, > > In case Open MPI is moved to a different location than it was installed into > initially, one has to export OPAL_PREFIX. While checking for the availability > of the GridEngine integration, I exported OPAL_PREFIX but obviously with a > typo and came to the conclusion that it's not available, as I did a: > > $ export PATH=… > $ export LD_LIBRARY_PATH=… > $ export OPAL_PREFIX=anything_with_a_typo_or_nothing_at_all > $ ompi_info | grep grid > > There was no indication, that `ompi_info` listed only part of the usual > output because of the faulty OPAL_PREFIX. When I recheck now, even the exit > code of `ompi_info` is still 0 in this case. > > I suggest to include some tests in `ompi_info` whether the set OPAL_PREFIX > makes sense. For now it lists just the set value in the "Prefix:" line and > that's all. > > Expected behavior: If `ompi_info` can't find any modules in the specified > place, an appropriate output should go to stderr and the exit code set to 1. No one commented on this. As the purpose of `ompi_info` is to check the installation (according to its man page), something should really be improved here. It's even possible to have both a 2.0.2 and 2.1.0 installed but only one in the $PATH of course, which may lead to: $ ~/local/openmpi-2.0.2_gcc-4.7.2_shared/bin/ompi_info -V Open MPI v2.0.2 Looks fine, but getting the full output: $ ~/local/openmpi-2.0.2_gcc-4.7.2_shared/bin/ompi_info Package: Open MPI reuti Distribution Open MPI: 2.1.0 shows a different Open MPI version as `mpiexec` and alike are in the actual set $PATH (not 2.0.2) (the former seems to be a constant in `ompi_info`, while the latter comes from a loaded library). Having a static version it's different: $ ~/local/openmpi-2.0.2_gcc-4.7.2_static/bin/ompi_info -V Open MPI v2.0.2 $ ~/local/openmpi-2.0.2_gcc-4.7.2_static/bin/ompi_info Package: Open MPI reuti Distribution Open MPI: 2.0.2 - -- Reuti -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAlkHa38ACgkQo/GbGkBRnRredgCeNmACv+HyJX2ERsltN+U+Zvs5 1wwAn22+zQBQVcf5d19zPxLbhqq81fqh =fd5O -END PGP SIGNATURE- ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] Behavior of `ompi_info`
Hi, In case Open MPI is moved to a different location than it was installed into initially, one has to export OPAL_PREFIX. While checking for the availability of the GridEngine integration, I exported OPAL_PREFIX but obviously with a typo and came to the conclusion that it's not available, as I did a: $ export PATH=… $ export LD_LIBRARY_PATH=… $ export OPAL_PREFIX=anything_with_a_typo_or_nothing_at_all $ ompi_info | grep grid There was no indication, that `ompi_info` listed only part of the usual output because of the faulty OPAL_PREFIX. When I recheck now, even the exit code of `ompi_info` is still 0 in this case. I suggest to include some tests in `ompi_info` whether the set OPAL_PREFIX makes sense. For now it lists just the set value in the "Prefix:" line and that's all. Expected behavior: If `ompi_info` can't find any modules in the specified place, an appropriate output should go to stderr and the exit code set to 1. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] openmpi-2.0.2
Due to the last post in this thread this copy I suggested seems not to be
possible, but I also want to test whether this post goes through to the list
now.
-- Reuti
===
Hi,
> Am 19.04.2017 um 19:53 schrieb Jim Edwards :
>
> Hi,
>
> I have openmpi-2.0.2 builds on two different machines and I have a test code
> which works on one machine and does not on the other machine. I'm struggling
> to understand why and I hope that by posting here someone may have some
> insight.
>
> The test is using mpi derived data types and mpi_alltoallw on 4 tasks. On
> the machine that fails it appears to ignore the displacement in the derived
> datatype defined on task 0 and just send 0-3 to all tasks.The failing
> machine is built against gcc 5.4.0, the working machine has both intel 16.0.3
> and gcc 6.3.0 builds.
And what happens when you copy one compilation from one node to the other
(including all addressed shared libraries)?
-- Reuti
> #include "mpi.h"
>
> #include
>
>
>
> int main(int argc, char *argv[])
>
> {
>
> int rank, size;
>
> MPI_Datatype type[4], type2[4];
>
> int displacement[1];
>
> int sbuffer[16];
>
> int rbuffer[4];
>
> MPI_Status status;
>
> int scnts[4], sdispls[4], rcnts[4], rdispls[4];
>
>
> MPI_Init(&argc, &argv);
>
> MPI_Comm_size(MPI_COMM_WORLD, &size);
>
> if (size < 4)
>
> {
>
> printf("Please run with 4 processes.\n");
>
> MPI_Finalize();
>
> return 1;
>
> }
>
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
>
>
> /* task 0 has sbuffer of size 16 and we are going to send 4 values to each
> of tasks 0-3, offsetting in each
>
> case so that the expected result is
>
> task[0] 0-3
>
> task[1] 4-7
>
> task[2] 8-11
>
> task[3] 12-15
>
> */
>
>
>
>
>
> for( int i=0; i
>if (rank == 0){
>
> scnts[i] = 1;
>
>}else{
>
> scnts[i] = 0;
>
>}
>
>sdispls[i] = 0;
>
>rcnts[i] = 0;
>
>rdispls[i] = 0;
>
> }
>
> rcnts[0] = 1;
>
>
> for (int i=0; i
>type[i] = MPI_INT;
>
>type2[i] = MPI_INT;
>
>rbuffer[i] = -1;
>
> }
>
>/* on the recv side we create a data type which is a single block of 4
> integers for the recv from 0
>
> otherwise we use MPI_INT as a placeholder for the type
>
> (openmpi does not want us to use MPI_DATATYPE_NULL a stupid misinterpretation
> of the standard imho )*/
>
>
>
> displacement[0] = 0;
>
> MPI_Type_create_indexed_block(1, 4, displacement, MPI_INT, type2);
>
> MPI_Type_commit(type2);
>
>
>
> if (rank == 0)
>
> {
>
>for( int i=0; i
> displacement[0] = i*4;
>
> /* we create a datatype which is a single block of 4 integers with offset 4
> from the start of sbuffer */
>
> MPI_Type_create_indexed_block(1, 4, displacement, MPI_INT, type + i);
>
> MPI_Type_commit(type+i);
>
>}
>
>for (int i=0; i<16; i++)
>
> sbuffer[i] = i;
>
> }
>
>
>
> for (int i=0; i
>printf("rank %d i=%d: scnts %d sdispls %d stype %d rcnts %d rdispls %d
> rtype %d\n", rank, i, scnts[i], sdispls[i], type[i], rcnts[i], rdispls[i],
> type2[i]);
>
>
> MPI_Alltoallw(sbuffer, scnts, sdispls, type, rbuffer, rcnts, rdispls, type2,
> MPI_COMM_WORLD);
>
>
>
> for (int i=0; i<4; i++)
>
>printf("rbuffer[%d] = %d\n", i, rbuffer[i]);
>
> fflush(stdout);
>
>
>
> MPI_Finalize();
>
> return 0;
>
> }
>
>
> --
> Jim Edwards
>
> CESM Software Engineer
> National Center for Atmospheric Research
> Boulder, CO
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Run-time issues with openmpi-2.0.2 and gcc
Hi, > Am 13.04.2017 um 11:00 schrieb Vincent Drach : > > > Dear mailing list, > > We are experimenting run time failure on a small cluster with openmpi-2.0.2 > and gcc 6.3 and gcc 5.4. > The job start normally and lots of communications are performed. After 5-10 > minutes the connection to the hosts is closed and > the following error message is reported: > -- > ORTE was unable to reliably start one or more daemons. > This usually is caused by: > > * not finding the required libraries and/or binaries on > one or more nodes. Please check your PATH and LD_LIBRARY_PATH > settings, or configure OMPI with --enable-orterun-prefix-by-default > > * lack of authority to execute on one or more specified nodes. > Please verify your allocation and authorities. > > * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base). > Please check with your sys admin to determine the correct location to use. > > * compilation of the orted with dynamic libraries when static are required > (e.g., on Cray). Please check your configure cmd line and consider using > one of the contrib/platform definitions for your system type. > > * an inability to create a connection back to mpirun due to a > lack of common network interfaces and/or no route found between > them. Please check network connectivity (including firewalls > and network routing requirements). > > > > The issue does not seem to be due to the infiniband configuration, because > the job also crash when using tcp protocol. > > Do you have any clue of what could be the issue ? Is it a single MPI process or is the application issuing many `mpiexec` during its runtime? Is there any limit how often `ssh` may access a node in a timeframe? Do you use any queuing system? -- Reuti signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No more default core binding since 2.0.2?
> Am 10.04.2017 um 17:27 schrieb r...@open-mpi.org: > > >> On Apr 10, 2017, at 1:37 AM, Reuti wrote: >> >>> >>> Am 10.04.2017 um 01:58 schrieb r...@open-mpi.org: >>> >>> Let me try to clarify. If you launch a job that has only 1 or 2 processes >>> in it (total), then we bind to core by default. This is done because a job >>> that small is almost always some kind of benchmark. >> >> Yes, I see. But only if libnuma was compiled in AFAICS. >> >> >>> If there are more than 2 processes in the job (total), then we default to >>> binding to NUMA (if NUMA’s are present - otherwise, to socket) across the >>> entire job. >> >> Mmh - can I spot a difference in --report-bindings between these two? To me >> both looks like being bound to socket. > > You won’t see a difference if the NUMA and socket are identical in terms of > the cores they cover. Ok, thx. >> >> -- Reuti >> >> >>> You can always override these behaviors. >>> >>>> On Apr 9, 2017, at 3:45 PM, Reuti wrote: >>>> >>>>>> But I can't see a binding by core for number of processes <= 2. Does it >>>>>> mean 2 per node or 2 overall for the `mpiexec`? >>>>> >>>>> It’s 2 processes overall >>>> >>>> Having a round-robin allocation in the cluster, this might not be what was >>>> intended (to bind only one or two cores per exechost)? >>>> >>>> Obviously the default changes (from --bind-to core to --bin-to socket), >>>> whether I compiled Open MPI with or w/o libnuma (I wanted to get rid of >>>> the warning in the output only – now it works). But "--bind-to core" I >>>> could also use w/o libnuma and it worked, I got only that warning in >>>> addition about the memory couldn't be bound. >>>> >>>> BTW: I always had to use -ldl when using `mpicc`. Now, that I compiled in >>>> libnuma, this necessity is gone. >>>> >>>> -- Reuti >>>> ___ >>>> users mailing list >>>> users@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >>> >>> ___ >>> users mailing list >>> users@lists.open-mpi.org >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >>> >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No more default core binding since 2.0.2?
> Am 10.04.2017 um 00:45 schrieb Reuti : > […]BTW: I always had to use -ldl when using `mpicc`. Now, that I compiled in > libnuma, this necessity is gone. Looks like I compiled too many versions in the last couple of days. The -ldl is necessary in case --disable-shared --enable-static was given to have a plain static version. -- Reuti signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No more default core binding since 2.0.2?
> Am 10.04.2017 um 01:58 schrieb r...@open-mpi.org: > > Let me try to clarify. If you launch a job that has only 1 or 2 processes in > it (total), then we bind to core by default. This is done because a job that > small is almost always some kind of benchmark. Yes, I see. But only if libnuma was compiled in AFAICS. > If there are more than 2 processes in the job (total), then we default to > binding to NUMA (if NUMA’s are present - otherwise, to socket) across the > entire job. Mmh - can I spot a difference in --report-bindings between these two? To me both looks like being bound to socket. -- Reuti > You can always override these behaviors. > >> On Apr 9, 2017, at 3:45 PM, Reuti wrote: >> >>>> But I can't see a binding by core for number of processes <= 2. Does it >>>> mean 2 per node or 2 overall for the `mpiexec`? >>> >>> It’s 2 processes overall >> >> Having a round-robin allocation in the cluster, this might not be what was >> intended (to bind only one or two cores per exechost)? >> >> Obviously the default changes (from --bind-to core to --bin-to socket), >> whether I compiled Open MPI with or w/o libnuma (I wanted to get rid of the >> warning in the output only – now it works). But "--bind-to core" I could >> also use w/o libnuma and it worked, I got only that warning in addition >> about the memory couldn't be bound. >> >> BTW: I always had to use -ldl when using `mpicc`. Now, that I compiled in >> libnuma, this necessity is gone. >> >> -- Reuti >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No more default core binding since 2.0.2?
>> But I can't see a binding by core for number of processes <= 2. Does it mean >> 2 per node or 2 overall for the `mpiexec`? > > It’s 2 processes overall Having a round-robin allocation in the cluster, this might not be what was intended (to bind only one or two cores per exechost)? Obviously the default changes (from --bind-to core to --bin-to socket), whether I compiled Open MPI with or w/o libnuma (I wanted to get rid of the warning in the output only – now it works). But "--bind-to core" I could also use w/o libnuma and it worked, I got only that warning in addition about the memory couldn't be bound. BTW: I always had to use -ldl when using `mpicc`. Now, that I compiled in libnuma, this necessity is gone. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] No more default core binding since 2.0.2?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Am 09.04.2017 um 16:35 schrieb r...@open-mpi.org: > There has been no change in the policy - however, if you are oversubscribed, > we did fix a bug to ensure that we don’t auto-bind in that situation > > Can you pass along your cmd line? So far as I can tell, it still seems to be > working. I'm not sure whether it was the case with 1.8, but according to the man page it binds now to sockets for number of processes > 2 . And this can lead the effect that one sometimes may notice a drop in performance when just this socket has other jobs running (by accident). So, this is solved - I wasn't aware of the binding by socket. But I can't see a binding by core for number of processes <= 2. Does it mean 2 per node or 2 overall for the `mpiexec`? - -- Reuti > >> On Apr 9, 2017, at 3:40 AM, Reuti wrote: >> >> Hi, >> >> While I noticed an automatic core binding in Open MPI 1.8 (which in a shared >> cluster may lead to oversubscribing of cores), I can't spot this any longer >> in the 2.x series. So the question arises: >> >> - Was this a general decision to no longer enable automatic core binding? >> >> First I thought it might be because of: >> >> - We define plm_rsh_agent=foo in $OMPI_ROOT/etc/openmpi-mca-params.conf >> - We compiled with --with-sge >> >> But also started on the command line by `ssh` to the nodes, there seems no >> automatic core binding to take place any longer. >> >> -- Reuti >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAljqnnYACgkQo/GbGkBRnRrwtACgpUAlpvQElzbjoVdvsQubZmTo Pj4An05kJd3pW0YWW4HXaf/7Zl7xTc+y =kzwG -END PGP SIGNATURE- ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] No more default core binding since 2.0.2?
Hi, While I noticed an automatic core binding in Open MPI 1.8 (which in a shared cluster may lead to oversubscribing of cores), I can't spot this any longer in the 2.x series. So the question arises: - Was this a general decision to no longer enable automatic core binding? First I thought it might be because of: - We define plm_rsh_agent=foo in $OMPI_ROOT/etc/openmpi-mca-params.conf - We compiled with --with-sge But also started on the command line by `ssh` to the nodes, there seems no automatic core binding to take place any longer. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] mpicc and libstdc++, general building question
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Am 07.04.2017 um 19:11 schrieb Christof Koehler: > […] > > On top, all OpenMPI libraries when checked with ldd (gcc 6.3 module > still loaded) reference the /usr/lib64/libstdc++.so.6 and not > /cluster/comp/gcc/6.3.0/lib64/libstdc++.so.6 which leads to the idea > that the OpenMPI installation might be the reason we have the > /usr/lib64/libstdc++.so.6 dependency in the mpi4py libraries as well. > > What we would like to have is that libstdc++.so.6 resolves to the > libstdc++ provided by the gcc 6.3 compiler for the mpi4py, which would be > available in its installation directory, i.e. > /cluster/comp/gcc/6.3.0/lib64/libstdc++.so.6. > > So, am I missing options in my OpenMPI build ? Should I explicitely do a > ./configure CC=/cluster/comp/gcc/6.3.0/bin/gcc > CXX=/cluster/comp/gcc/6.3.0/bin/g++ ... > or similar ? Am I building it correctly with a gcc contained in a > separate module anyway ? Or do we have a problem with our ld configuration ? I have a default GCC 4.7.2 in the system, and I just prepend PATH and LD_LIBRARY_PATH with the paths to my private GCC 6.2.0 compilation in my home directory by a plain `export`. I can spot: $ ldd libmpi_cxx.so.20 … libstdc++.so.6 => /home/reuti/local/gcc-6.2.0/lib64/../lib64/libstdc++.so.6 (0x7f184d2e2000) So this looks fine (although /lib64/../lib64/ looks nasty). In the library, the RPATH and RUNPATH are set: $ readelf -a libmpi_cxx.so.20 … 0x000f (RPATH) Library rpath: [/home/reuti/local/openmpi-2.1.0_gcc-6.2.0_shared/lib64:/home/reuti/local/gcc-6.2.0/lib64/../lib64] 0x0000001d (RUNPATH)Library runpath: [/home/reuti/local/openmpi-2.1.0_gcc-6.2.0_shared/lib64:/home/reuti/local/gcc-6.2.0/lib64/../lib64] Can you check the order in your PATH and LD_LIBRARY_PATH – are they as expected when loading the module? - -- Reuti -BEGIN PGP SIGNATURE- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAljn+KkACgkQo/GbGkBRnRq3wwCgkRkiaPyXBdAMHoABmFBfDevu ftkAnR3gul9AnZL0qqb8vZg8zjJvIHtR =M5Ya -END PGP SIGNATURE- ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Compiler error with PGI: pgcc-Error-Unknown switch: -pthread
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Am 03.04.2017 um 23:07 schrieb Prentice Bisbal:
> FYI - the proposed 'here-doc' solution below didn't work for me, it produced
> an error. Neither did printf. When I used printf, only the first arg was
> passed along:
>
> #!/bin/bash
>
> realcmd=/usr/pppl/pgi/17.3/linux86-64/17.3/bin/pgcc.real
> echo "original args: $@"
> newargs=$(printf -- "$@" | sed s/-pthread//g)
The format string is missing:
printf "%s " "$@"
> echo "new args: $newargs"
> #$realcmd $newargs
> exit
>
> $ pgcc -tp=x64 -fast conftest.c
> original args: -tp=x64 -fast conftest.c
> new args: -tp=x64
>
> Any ideas what I might be doing wrong here?
>
> So, my original echo "" "$@" solution works, and another colleague also
> suggested this expressions, which appears to work, too:
>
> newargs=${@/-pthread/}
>
> Although I don't know how portable that is. I'm guessing that's very
> bash-specific syntax.
>
> Prentice
>
> On 04/03/2017 04:26 PM, Prentice Bisbal wrote:
>> A coworker came up with another idea that works, too:
>>
>> newargs=sed s/-pthread//g <> $@
>> EOF
>>
>> That should work, too, but I haven't test it.
>>
>> Prentice
>>
>> On 04/03/2017 04:11 PM, Andy Riebs wrote:
>>> Try
>>> $ printf -- "-E" ...
>>>
>>> On 04/03/2017 04:03 PM, Prentice Bisbal wrote:
Okay. the additional -E doesn't work,either. :(
Prentice Bisbal Lead Software Engineer Princeton Plasma Physics Laboratory
http://www.pppl.gov
On 04/03/2017 04:01 PM, Prentice Bisbal wrote:
> Nevermind. A coworker helped me figure this one out. Echo is treating the
> '-E' as an argument to echo and interpreting it instead of passing it to
> sed. Since that's used by the configure tests, that's a bit of a problem,
> Just adding another -E before $@, should fix the problem.
>
> Prentice
>
> On 04/03/2017 03:54 PM, Prentice Bisbal wrote:
>> I've decided to work around this problem by creating a wrapper script
>> for pgcc that strips away the -pthread argument, but my sed expression
>> works on the command-line, but not in the script. I'm essentially
>> reproducing the workaround from
>> https://www.open-mpi.org/community/lists/users/2009/04/8724.php.
>>
>> Can anyone see what's wrong with my implementation the workaround? It's
>> a very simple sed expression. Here's my script:
>>
>> #!/bin/bash
>>
>> realcmd=/path/to/pgcc
>> echo "original args: $@"
>> newargs=$(echo "$@" | sed s/-pthread//)
>> echo "new args: $newargs"
>> #$realcmd $newargs
>> exit
>>
>> And here's what happens when I run it:
>>
>> /path/to/pgcc -E conftest.c
>> original args: -E conftest.c
>> new args: conftest.c
>>
>> As you can see, the -E argument is getting lost in translation. If I add
>> more arguments, it works fine:
>>
>> /path/to/pgcc -A -B -C -D -E conftest.c
>> original args: -A -B -C -D -E conftest.c
>> new args: -A -B -C -D -E conftest.c
>>
>> It only seems to be a problem when -E is the first argument:
>>
>> $ /path/to/pgcc -E -D -C -B -A conftest.c
>> original args: -E -D -C -B -A conftest.c
>> new args: -D -C -B -A conftest.c
>>
>> Prentice
>>
>> On 04/03/2017 02:24 PM, Aaron Knister wrote:
>>> To be thorough couldn't one replace -pthread in the slurm .la files
>>> with -lpthread? I ran into this last week and this was the solution I
>>> was thinking about implementing. Having said that, I can't think of a
>>> situation in which the -pthread/-lpthread argument would be required
>>> other than linking against statically compiled SLURM libraries and even
>>> then I'm not so sure about that.
>>>
>>> -Aaron
>>>
>>> On 4/3/17 1:46 PM, �ke Sandgren wrote:
We build slurm with GCC, drop the -pthread arg in the .la files, and
have never seen any problems related to that. And we do build quite a
lot of code. And lots of versions of OpenMPI with multiple different
compilers (and versions).
On 04/03/2017 04:51 PM, Prentice Bisbal wrote:
> This is the second suggestion to rebuild Slurm
>
> The other from �ke Sandgren, who recommended this:
>
>> This usually comes from slurm, so we always do
>>
>> perl -pi -e 's/-pthread//' /lap/slurm/${version}/lib/libpmi.la
>> /lap/slurm/${version}/lib/libslurm.la
>>
>> when installing a new slurm version. Thus no need for a fakepg
>> wrapper.
>
> I don't really have the luxury to rebuild Slurm at the moment. How
> would
> I rebuild Slurm to change this behavior? Is rebuilding Slurm with PGI
> the only option to fix this in slurm, or use �ke's suggestion above?
>
> If I did use �ke's suggestion above, how would that affect t
Re: [OMPI users] Compiler error with PGI: pgcc-Error-Unknown switch: -pthread
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Am 03.04.2017 um 22:01 schrieb Prentice Bisbal:
> Nevermind. A coworker helped me figure this one out. Echo is treating the
> '-E' as an argument to echo and interpreting it instead of passing it to sed.
> Since that's used by the configure tests, that's a bit of a problem, Just
> adding another -E before $@, should fix the problem.
It's often suggested to use printf instead of the non-portable echo.
- -- Reuti
>
> Prentice
>
> On 04/03/2017 03:54 PM, Prentice Bisbal wrote:
>> I've decided to work around this problem by creating a wrapper script for
>> pgcc that strips away the -pthread argument, but my sed expression works on
>> the command-line, but not in the script. I'm essentially reproducing the
>> workaround from
>> https://www.open-mpi.org/community/lists/users/2009/04/8724.php.
>>
>> Can anyone see what's wrong with my implementation the workaround? It's a
>> very simple sed expression. Here's my script:
>>
>> #!/bin/bash
>>
>> realcmd=/path/to/pgcc
>> echo "original args: $@"
>> newargs=$(echo "$@" | sed s/-pthread//)
>> echo "new args: $newargs"
>> #$realcmd $newargs
>> exit
>>
>> And here's what happens when I run it:
>>
>> /path/to/pgcc -E conftest.c
>> original args: -E conftest.c
>> new args: conftest.c
>>
>> As you can see, the -E argument is getting lost in translation. If I add
>> more arguments, it works fine:
>>
>> /path/to/pgcc -A -B -C -D -E conftest.c
>> original args: -A -B -C -D -E conftest.c
>> new args: -A -B -C -D -E conftest.c
>>
>> It only seems to be a problem when -E is the first argument:
>>
>> $ /path/to/pgcc -E -D -C -B -A conftest.c
>> original args: -E -D -C -B -A conftest.c
>> new args: -D -C -B -A conftest.c
>>
>> Prentice
>>
>> On 04/03/2017 02:24 PM, Aaron Knister wrote:
>>> To be thorough couldn't one replace -pthread in the slurm .la files with
>>> -lpthread? I ran into this last week and this was the solution I was
>>> thinking about implementing. Having said that, I can't think of a situation
>>> in which the -pthread/-lpthread argument would be required other than
>>> linking against statically compiled SLURM libraries and even then I'm not
>>> so sure about that.
>>>
>>> -Aaron
>>>
>>> On 4/3/17 1:46 PM, Åke Sandgren wrote:
>>>> We build slurm with GCC, drop the -pthread arg in the .la files, and
>>>> have never seen any problems related to that. And we do build quite a
>>>> lot of code. And lots of versions of OpenMPI with multiple different
>>>> compilers (and versions).
>>>>
>>>> On 04/03/2017 04:51 PM, Prentice Bisbal wrote:
>>>>> This is the second suggestion to rebuild Slurm
>>>>>
>>>>> The other from Åke Sandgren, who recommended this:
>>>>>
>>>>>> This usually comes from slurm, so we always do
>>>>>>
>>>>>> perl -pi -e 's/-pthread//' /lap/slurm/${version}/lib/libpmi.la
>>>>>> /lap/slurm/${version}/lib/libslurm.la
>>>>>>
>>>>>> when installing a new slurm version. Thus no need for a fakepg wrapper.
>>>>>
>>>>> I don't really have the luxury to rebuild Slurm at the moment. How would
>>>>> I rebuild Slurm to change this behavior? Is rebuilding Slurm with PGI
>>>>> the only option to fix this in slurm, or use Åke's suggestion above?
>>>>>
>>>>> If I did use Åke's suggestion above, how would that affect the operation
>>>>> of Slurm, or future builds of OpenMPI and any other software that might
>>>>> rely on Slurm, particulary with regards to building those apps with
>>>>> non-PGI compilers?
>>>>>
>>>>> Prentice
>>>>>
>>>>> On 04/03/2017 10:31 AM, Gilles Gouaillardet wrote:
>>>>>> Hi,
>>>>>>
>>>>>> The -pthread flag is likely pulled by libtool from the slurm libmpi.la
>>>>>> <http://libmpi.la> and/or libslurm.la <http://libslurm.la>
>>>>>> Workarounds are
>>>>>> - rebuild slurm with PGI
>>>>>> - remove the .la files (*.so and/or *.a are enough)
>>>>>> - wrap the PGI compiler to ignore the -pthread option
>>
Re: [OMPI users] Performance degradation of OpenMPI 1.10.2 when oversubscribed?
Hi, Am 24.03.2017 um 20:39 schrieb Jeff Squyres (jsquyres): > Limiting MPI processes to hyperthreads *helps*, but current generation Intel > hyperthreads are not as powerful as cores (they have roughly half the > resources of a core), so -- depending on your application and your exact > system setup -- you will almost certainly see performance degradation of > running N MPI processes across N cores vs. across N hyper threads. You can > try it yourself by running the same size application over N cores on a single > machine, and then run the same application over N hyper threads (i.e., N/2 > cores) on the same machine. > > […] > > - Disabling HT in the BIOS means that the one hardware thread left in each > core will get all the cores resources (buffers, queues, processor units, > etc.). > - Enabling HT in the BIOS means that each of the 2 hardware threads will > statically be allocated roughly half the core's resources (buffers, queues, > processor units, etc.). Do you have a reference for the two topics above (sure, I will try next week on my own)? My knowledge was, that there is no dedicated HT core, and using all cores will not give the result that the real cores get N x 100%, plus the HT ones N x 50% (or alike). But the scheduler inside the CPU will balance the resources between the double face of a single core and both are equal. > […] > Spoiler alert: many people have looked at this. In *most* (but not all) > cases, using HT is not a performance win for MPI/HPC codes that are designed > to run processors at 100%. I think it was also on this mailing list, that someone mentioned that the pipelines in the CPU are reorganized in case you switch HT off, as only half of them would be needed and these resources are then bound to the real cores too, extending their performance. Similar, but not exactly what Jeff mentiones above. Another aspect is, that even if they are not really doubling the performance, one might get 150%. And if you pay per CPU hours, it can be worth to have it switched on. My personal experience is, that it depends not only application, but also on the way how you oversubscribe. Using all cores for a single MPI application leads to the effect, that all processes are doing the same stuff at the same time (at least often) and fight for the same part of the CPU, essentially becoming a bottleneck. But using each half of a CPU for two (or even more) applications will allow a better interleaving in the demand for resources. To allow this in the best way: no taskset or binding to cores, let the Linux kernel and CPU do their best - YMMV. -- Reuti ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Help with Open MPI 2.1.0 and PGI 16.10: Configure and C++
Hi, Am 22.03.2017 um 20:12 schrieb Matt Thompson: > […] > > Ah. PGI 16.9+ now use pgc++ to do C++ compiling, not pgcpp. So, I hacked > configure so that references to pgCC (nonexistent on macOS) are gone and all > pgcpp became pgc++, but: This is not unique to macOS. pgCC used STLPort STL and is no longer included with their compiler suite, pgc++ now uses a GCC compatible library format and replaces the former one on Linux too. There I get, ignoring the gnu output during `configure` and compiling anyway: $ mpic++ --version pgc++ 16.10-0 64-bit target on x86-64 Linux -tp bulldozer The Portland Group - PGI Compilers and Tools Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. Maybe some options for the `mpic++` wrapper were just set in a wrong way? === Nevertheless: did you see the error on the Mac at the end of the `configure` step too, or was it gone after the hints on the discussion's link you posted? As I face it there still about "libeevent". -- Reuti > > *** C++ compiler and preprocessor > checking whether we are using the GNU C++ compiler... yes > checking whether pgc++ accepts -g... yes > checking dependency style of pgc++... none > checking how to run the C++ preprocessor... pgc++ -E > checking for the C++ compiler vendor... gnu > > Well, at this point, I think I'm stopping until I get help. Will this chunk > of configure always return gnu for PGI? I know the C part returns 'portland > group': > > *** C compiler and preprocessor > checking for gcc... (cached) pgcc > checking whether we are using the GNU C compiler... (cached) no > checking whether pgcc accepts -g... (cached) yes > checking for pgcc option to accept ISO C89... (cached) none needed > checking whether pgcc understands -c and -o together... (cached) yes > checking for pgcc option to accept ISO C99... none needed > checking for the C compiler vendor... portland group > > so I thought the C++ section would as well. I also tried passing in > --enable-mpi-cxx, but that did nothing. > > Is this just a red herring? My real concern is with pgfortran/mpifort, but I > thought I'd start with this. If this is okay, I'll move on and detail the > fortran issues I'm having. > > Matt > -- > Matt Thompson > Man Among Men > Fulcrum of History > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OpenMPI-2.1.0 problem with executing orted when using SGE
> Am 22.03.2017 um 15:31 schrieb Heinz-Ado Arnolds > : > > Dear Reuti, > > thanks a lot, you're right! But why did the default behavior change but not > the value of this parameter: > > 2.1.0: MCA plm rsh: parameter "plm_rsh_agent" (current value: "ssh : rsh", > data source: default, level: 2 user/detail, type: string, synonyms: > pls_rsh_agent, orte_rsh_agent) > The command used to launch executables on remote > nodes (typically either "ssh" or "rsh") > > 1.10.6: MCA plm: parameter "plm_rsh_agent" (current value: "ssh : rsh", data > source: default, level: 2 user/detail, type: string, synonyms: pls_rsh_agent, > orte_rsh_agent) > The command used to launch executables on remote > nodes (typically either "ssh" or "rsh") > > That means there must have been changes in the code regarding that, perhaps > for detecting SGE? Do you know of a way to revert to the old style (e.g. > configure option)? Otherwise all my users have to add this option. There was a discussion in https://github.com/open-mpi/ompi/issues/2947 For now you can make use of https://www.open-mpi.org/faq/?category=tuning#setting-mca-params Essentially to have it set for all users automatically, put: plm_rsh_agent=foo in $prefix/etc/openmpi-mca-params.conf of your central Open MPI 2.1.0 installation. -- Reuti > Thanks again, and have a nice day > > Ado Arnolds > > On 22.03.2017 13:58, Reuti wrote: >> Hi, >> >>> Am 22.03.2017 um 10:44 schrieb Heinz-Ado Arnolds >>> : >>> >>> Dear users and developers, >>> >>> first of all many thanks for all the great work you have done for OpenMPI! >>> >>> Up to OpenMPI-1.10.6 the mechanism for starting orted was to use SGE/qrsh: >>> mpirun -np 8 --map-by ppr:4:node ./myid >>> /opt/sge-8.1.8/bin/lx-amd64/qrsh -inherit -nostdin -V >> Machine> orted --hnp-topo-sig 2N:2S:2L3:20L2:20L1:20C:40H:x86_64 -mca ess >>> "env" -mca orte_ess_jobid "1621884928" -mca orte_ess_vpid 1 -mca >>> orte_ess_num_procs "2" -mca orte_hnp_uri "1621884928.0;tcp://>> Master>:41031" -mca plm "rsh" -mca rmaps_base_mapping_policy "ppr:4:node" >>> --tree-spawn >>> >>> Now with OpenMPI-2.1.0 (and the release candidates) "ssh" is used to start >>> orted: >>> mpirun -np 8 --map-by ppr:4:node -mca mca_base_env_list OMP_NUM_THREADS=5 >>> ./myid >>> /usr/bin/ssh -x >>> PATH=/afs/./openmpi-2.1.0/bin:$PATH ; export PATH ; >>> LD_LIBRARY_PATH=/afs/./openmpi-2.1.0/lib:$LD_LIBRARY_PATH ; export >>> LD_LIBRARY_PATH ; >>> DYLD_LIBRARY_PATH=/afs/./openmpi-2.1.0/lib:$DYLD_LIBRARY_PATH ; export >>> DYLD_LIBRARY_PATH ; /afs/./openmpi-2.1.0/bin/orted --hnp-topo-sig >>> 2N:2S:2L3:20L2:20L1:20C:40H:x86_64 -mca ess "env" -mca ess_base_jobid >>> "1626013696" -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca >>> orte_hnp_uri "1626013696.0;usock;tcp://:43019" -mca >>> plm_rsh_args "-x" -mca plm "rsh" -mca rmaps_base_mapping_policy >>> "ppr:4:node" -mca pmix "^s1,s2,cray" >>> >>> qrsh set the environment properly on the remote side, so that environment >>> variables from job scripts are properly transferred. With the ssh variant >>> the environment is not set properly on the remote side, and it seems that >>> there are handling problems with Kerberos tickets and/or AFS tokens. >>> >>> Is there any way to revert the 2.1.0 behavior to the 1.10.6 (use SGE/qrsh) >>> one? Are there mca params to set this? >>> >>> If you need more info, please let me know. (Job submitting machine and >>> target cluster are the same with all tests. SW is residing in AFS >>> directories visible on all machines. Parameter "plm_rsh_disable_qrsh" >>> current value: "false") >> >> It looks like `mpirun` still needs: >> >> -mca plm_rsh_agent foo >> >> to allow SGE to be detected. >> >> -- Reuti >> >> >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OpenMPI-2.1.0 problem with executing orted when using SGE
Hi, > Am 22.03.2017 um 10:44 schrieb Heinz-Ado Arnolds > : > > Dear users and developers, > > first of all many thanks for all the great work you have done for OpenMPI! > > Up to OpenMPI-1.10.6 the mechanism for starting orted was to use SGE/qrsh: > mpirun -np 8 --map-by ppr:4:node ./myid > /opt/sge-8.1.8/bin/lx-amd64/qrsh -inherit -nostdin -V Machine> orted --hnp-topo-sig 2N:2S:2L3:20L2:20L1:20C:40H:x86_64 -mca ess > "env" -mca orte_ess_jobid "1621884928" -mca orte_ess_vpid 1 -mca > orte_ess_num_procs "2" -mca orte_hnp_uri "1621884928.0;tcp:// Master>:41031" -mca plm "rsh" -mca rmaps_base_mapping_policy "ppr:4:node" > --tree-spawn > > Now with OpenMPI-2.1.0 (and the release candidates) "ssh" is used to start > orted: > mpirun -np 8 --map-by ppr:4:node -mca mca_base_env_list OMP_NUM_THREADS=5 > ./myid > /usr/bin/ssh -x > PATH=/afs/./openmpi-2.1.0/bin:$PATH ; export PATH ; > LD_LIBRARY_PATH=/afs/./openmpi-2.1.0/lib:$LD_LIBRARY_PATH ; export > LD_LIBRARY_PATH ; > DYLD_LIBRARY_PATH=/afs/./openmpi-2.1.0/lib:$DYLD_LIBRARY_PATH ; export > DYLD_LIBRARY_PATH ; /afs/./openmpi-2.1.0/bin/orted --hnp-topo-sig > 2N:2S:2L3:20L2:20L1:20C:40H:x86_64 -mca ess "env" -mca ess_base_jobid > "1626013696" -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca > orte_hnp_uri "1626013696.0;usock;tcp://:43019" -mca > plm_rsh_args "-x" -mca plm "rsh" -mca rmaps_base_mapping_policy "ppr:4:node" > -mca pmix "^s1,s2,cray" > > qrsh set the environment properly on the remote side, so that environment > variables from job scripts are properly transferred. With the ssh variant the > environment is not set properly on the remote side, and it seems that there > are handling problems with Kerberos tickets and/or AFS tokens. > > Is there any way to revert the 2.1.0 behavior to the 1.10.6 (use SGE/qrsh) > one? Are there mca params to set this? > > If you need more info, please let me know. (Job submitting machine and target > cluster are the same with all tests. SW is residing in AFS directories > visible on all machines. Parameter "plm_rsh_disable_qrsh" current value: > "false") It looks like `mpirun` still needs: -mca plm_rsh_agent foo to allow SGE to be detected. -- Reuti signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] State of the DVM in Open MPI
Hi, Only by reading recent posts I got aware of the DVM. This would be a welcome feature for our setup*. But I see not all options working as expected - is it still a work in progress, or should all work as advertised? 1) $ soft@server:~> orte-submit -cf foo --hnp file:/home/reuti/dvmuri -n 1 touch /home/reuti/hacked Open MPI has detected that a parameter given to a command line option does not match the expected format: Option: np Param: foo ==> The given option is -cf, not -np 2) According to `man orte-dvm` there is -H, -host, --host, -machinefile, -hostfile but none of them seem operational (Open MPI 2.0.2). A given hostlist given by SGE is honored though. -- Reuti *) We run Open MPI jobs inside SGE. This works fine. Some applications invoke several `mpiexec`-calls during their execution and rely on temporary files they created in the last step(s). While this is working fine on one and the same machine, it fails in case SGE granted slots on several machines as the scratch directories created by `qrsh -inherit …` vanish once the `mpiexec`-call on this particular node finishes (and not at the end of the complete job). I can mimic persistent scratch directories in SGE for a complete job, but invoking the DVM before and shutting it down later on (either by hand in the job script or by SGE killing all remains at the end of the job) might be more straight forward (looks like `orte-dvm` is started by `qrsh -inherit …` too). signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel
> Am 27.02.2017 um 18:24 schrieb Angel de Vicente : > > […] > > For a small group of users if the DVM can run with my user and there is > no restriction on who can use it or if I somehow can authorize others to > use it (via an authority file or similar) that should be enough. AFAICS there is no user authorization at all. Everyone can hijack a running DVM once he knows the URI. The only problem might be, that all processes are running under the account of the user who started the DVM. I.e. output files have to go to the home directory of this user, as any other user can't write to his own directory any longer this way. Running the DVM under root might help, but this would be a high risk that any faulty script might write to a place where sensible system information is stored and may leave the machine unusable afterwards. My first attempts using DVM often leads to a terminated DVM once a process returned with a non-zero exit code. But once the DVM is gone, the queued jobs might be lost too I fear. I would wish that the DVM could be more forgivable (or this feature be adjustable what to do in case of a non-zero exit code). -- Reuti signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Using OpenMPI / ORTE as cluster aware GNU Parallel
Hi, > Am 27.02.2017 um 14:33 schrieb Angel de Vicente : > > Hi, > > "r...@open-mpi.org" writes: >>> With the DVM, is it possible to keep these jobs in some sort of queue, >>> so that they will be executed when the cores get free? >> >> It wouldn’t be hard to do so - as long as it was just a simple FIFO >> scheduler. I wouldn’t want it to get too complex. > > a simple FIFO should be probably enough. This can be useful as a simple > way to make a multi-core machine accessible to a small group of (friendly) > users, making sure that they don't oversubscribe the machine, but > without going the full route of installing/maintaining a full resource > manager. At first I thought you want to run a queuing system inside a queuing system, but this looks like you want to replace the resource manager. Under which user account the DVM daemons will run? Are all users using the same account? -- Reuti signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Is gridengine integration broken in openmpi 2.0.2?
Hi, > Am 03.02.2017 um 17:10 schrieb Mark Dixon : > > Hi, > > Just tried upgrading from 2.0.1 to 2.0.2 and I'm getting error messages that > look like openmpi is using ssh to login to remote nodes instead of qrsh (see > below). Has anyone else noticed gridengine integration being broken, or am I > being dumb? > > I built with "./configure > --prefix=/apps/developers/libraries/openmpi/2.0.2/1/intel-17.0.1 --with-sge > --with-io-romio-flags=--with-file-system=lustre+ufs --enable-mpi-cxx > --with-cma" SGE on its own is not configured to use SSH? (I mean the entries in `qconf -sconf` for rsh_command resp. daemon). -- Reuti > Can see the gridengine component via: > > $ ompi_info -a | grep gridengine > MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component v2.0.2) > MCA ras gridengine: --- > MCA ras gridengine: parameter "ras_gridengine_priority" (current value: > "100", data source: default, level: 9 dev/all, type: int) > Priority of the gridengine ras component > MCA ras gridengine: parameter "ras_gridengine_verbose" (current value: > "0", data source: default, level: 9 dev/all, type: int) > Enable verbose output for the gridengine ras > component > MCA ras gridengine: parameter "ras_gridengine_show_jobid" (current > value: "false", data source: default, level: 9 dev/all, type: bool) > > Cheers, > > Mark > > ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory > Permission denied, please try again. > ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory > Permission denied, please try again. > ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory > Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased). > -- > ORTE was unable to reliably start one or more daemons. > This usually is caused by: > > * not finding the required libraries and/or binaries on > one or more nodes. Please check your PATH and LD_LIBRARY_PATH > settings, or configure OMPI with --enable-orterun-prefix-by-default > > * lack of authority to execute on one or more specified nodes. > Please verify your allocation and authorities. > > * the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base). > Please check with your sys admin to determine the correct location to use. > > * compilation of the orted with dynamic libraries when static are required > (e.g., on Cray). Please check your configure cmd line and consider using > one of the contrib/platform definitions for your system type. > > * an inability to create a connection back to mpirun due to a > lack of common network interfaces and/or no route found between > them. Please check network connectivity (including firewalls > and network routing requirements). > -- > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users signature.asc Description: Message signed with OpenPGP using GPGMail ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Low CPU utilization
Hi, Am 16.10.2016 um 20:34 schrieb Mahmood Naderan: > Hi, > I am running two softwares that use OMPI-2.0.1. Problem is that the CPU > utilization is low on the nodes. > > > For example, see the process information below > > [root@compute-0-1 ~]# ps aux | grep siesta > mahmood 14635 0.0 0.0 108156 1300 ?S21:58 0:00 /bin/bash > /share/apps/chemistry/siesta-4.0-mpi201/spar/siesta.p1 A.fdf > mahmood 14636 0.0 0.0 108156 1300 ?S21:58 0:00 /bin/bash > /share/apps/chemistry/siesta-4.0-mpi201/spar/siesta.p1 A.fdf > mahmood 14637 61.6 0.2 372076 158220 ? Rl 21:58 0:38 > /share/apps/chemistry/siesta-4.0-mpi201/spar/siesta > mahmood 14639 59.6 0.2 365992 154228 ? Rl 21:58 0:37 > /share/apps/chemistry/siesta-4.0-mpi201/spar/siesta > > > Note that the cpu utilization is the third column. The "siesta.pl" script is > > #!/bin/bash > BENCH=$1 > export OMP_NUM_THREADS=1 > /share/apps/chemistry/siesta-4.0-mpi201/spar/siesta < $BENCH > > > > > I also saw a similar behavior from Gromacs which has been discussed at > https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-October/108939.html > > It seems that there is a tricky thing with OMPI. Any idea is welcomed. Sounds like the two jobs are using the same cores by automatic core binding as one instance doesn't know anything about the other. For a first test you can start both with "mpiexec --bind-to none ..." and check whether you see a different behavior. `man mpiexec` mentions some hints about threads in applications. -- Reuti > > > Regards, > Mahmood > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] Running a computer on multiple computers
Am 14.10.2016 um 23:10 schrieb Mahdi, Sam: > Hello everyone, > > I am attempting to run a single program on 32 cores split across 4 computers > (So each computer has 8 cores). I am attempting to use mpich for this. I > currently am just testing on 2 computers, I have the program installed on > both, as well as mpich installed on both. I have created a register key and > can login in using ssh into the other computer without a password. I have > come across 2 problems. One, when I attempt to connect using the mpirun -np 3 > --host a (the IP of the computer I am attempting to connect to) hostname > I recieve the error > unable to connect from "localhost.localdomain" to "localhost.localdomain" > > This is indicating my computers "localhost.localdomain" is attempting to > connect to another "localhost.localdomain". How can I change this so that it > connects via my IP to the other computers IP? > > Secondly, I attempted to use a host file instead using the hydra process > wiki. I created a hosts file with just the IP of the computer I am attempting > to connect to. When I type in the command mpiexec -f hosts -n 4 ./applic > > I get this error > [mpiexec@localhost.localdomain] HYDU_parse_hostfile > (./utils/args/args.c:323): unable to open host file: hosts As you mentioned MPICH and their Hydra startup, you better ask at their list: http://www.mpich.org/support/mailing-lists/ This list is for the Open MPI implementation. -- Reuti > > along with other errors of unable to parse hostfile, match handler etc. I > assume this is all due to it being unable to read the host file. Is there any > specific place I should save my hosts file? I have it saved directly on my > Desktop. I have attempted to indicate the full path where it is located, but > I still get the same error. > > For the first problem, I have read that I need to change /etc/hosts manually > by using the sudo command to manually enter the IP of the computer I am > attempting to connect to in the /etc/hosts file. Thank you in advance. > > Sincerely, > Sam > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] MPI Behaviour Question
Hi, > Am 11.10.2016 um 14:56 schrieb Mark Potter : > > This question is related to OpenMPI 2.0.1 compiled with GCC 4.8.2 on > RHEL 6.8 using Torque 6.0.2 with Moab 9.0.2. To be clear, I am an > administrator and not a coder and I suspect this is expected behavior > but I have been asked by a client to explain why this is happening. > > Using Torque, the following command returns the hostname of the first > node only, regardless of how the nodes/cores are split up: > > mpirun -np 20 echo "Hello from $HOSTNAME" The $HOSTNAME will be expanded and used as argument before `mpirun` even starts. Instead it has to be evaluated on the nodes: $ mpirun bash -c "echo \$HOSTNAME" > (the behaviour is the same with "echo $(hostname)) > > The Torque script looks like this: > > #PBS -V > #PBS -N test-job > #PBS -l nodes=2:ppn=16 > #PBS -e ERROR > #PBS -o OUTPUT > > > cd $PBS_O_WORKDIR > date > cat $PBS_NODEFILE > > mpirun -np32 echo "Hello from $HOSTNAME" > > If the echo statement is replaced with "hostname" then a proper > response is received from all nodes. > > While I know there are better ways to test OpenMPI's functionality, > like compiling and using the programs in examples/, this is the method > a specific client chose. There are small "Hello world" programs like here: http://mpitutorial.com/tutorials/mpi-hello-world/ to test whether e.g. the libraries are found at runtime by the application(s). -- Reuti > I was using both the examples and a Torque job > script calling just "hostname" as a command and not using echo and the > client was using the script above. It took some doing to figure out why > he thought it wasn't working and all my tests were successful and when > I figured it, he wanted an explanation that's beyond my current > knowledge. Any help towards explaining the behaviour would be greatly > appreciated. > > -- > Regards, > > Mark L. Potter > Senior Consultant > PCPC Direct, Ltd. > O: 713-344-0952 > M: 713-965-4133 > S: mpot...@pcpcdirect.com > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OMPI users] Still "illegal instruction"
> Am 22.09.2016 um 17:20 schrieb Mahmood Naderan : > > Although this problem is not related to OMPI *at all*, I think it is good to > tell the others what was going on. Finally, I caught the illegal instruction > :) > > Briefly, I built the serial version of Siesta on the frontend and ran it > directly on the compute node. Fortunately, "x/i $pc" from GDB showed that the > illegal instruction was a FMA3 instruction. More detail is available at > https://gcc.gnu.org/ml/gcc-help/2016-09/msg00084.html > > According to the Wikipedia, > > • FMA4 is supported in AMD processors starting with the Bulldozer > architecture. FMA4 was realized in hardware before FMA3. > • FMA3 is supported in AMD processors starting with the Piledriver > architecture and Intel starting with Haswell processors and Broadwell > processors since 2014. > Therefore, the frontend (piledriver) inserts a FMA3 instruction while the > compute node (Bulldozer) doesn't recognize it. Thx for sharing, quite interesting. But does this mean, that there is no working command line flag for gcc to switch this off (like -march=bdver1 what Gilles mentioned) or to tell me what he thinks it should compile for? For pgcc there is -show and I can spot the target it discovered in the USETPVAL= line. -- Reuti > > The solution was (as stated by guys) building Siesta on the compute node. I > have to say that I tested all related programs (OMPI, Scalapack, OpenBLAS) > sequentially on the compute node in order to find who generate the illegal > instruction. > > Anyway... thanks a lot for your comments. Hope this helps others in the > future. > > > > Regards, > Mahmood > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OMPI users] Still "illegal instruction"
Am 15.09.2016 um 19:54 schrieb Mahmood Naderan: > The differences are very very minor > > root@cluster:tpar# echo | gcc -v -E - 2>&1 | grep cc1 > /usr/libexec/gcc/x86_64-redhat-linux/4.4.7/cc1 -E -quiet -v - -mtune=generic > > [root@compute-0-1 ~]# echo | gcc -v -E - 2>&1 | grep cc1 > /usr/libexec/gcc/x86_64-redhat-linux/4.4.6/cc1 -E -quiet -v - -mtune=generic > > > Even I tried to compile the program with -march=amdfam10. Something like these > > /export/apps/siesta/openmpi-2.0.0/bin/mpifort -c -g -Os -march=amdfam10 > `FoX/FoX-config --fcflags` -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT > -DTRANSIESTA/export/apps/siesta/siesta-4.0/Src/pspltm1.F > > But got the same error. > > /proc/cpuinfo on the frontend shows (family 21, model 2) and on the compute > node it shows (family 21, model 1). Just for curiosity: what is the model name of them? > >That being said, my best bet is you compile on a compute node ... > gcc is there on the computes, but the NFS permission is another issue. It > seems that nodes are not able to write on /share (the one which is shared > between frontend and computes). Would it work to compile with a shared target and copy it to /shared on the frontend? -- Reuti > An important question is that, how can I find out what is the name of the > illegal instruction. Then, I hope to find the document that points which > instruction set (avx, sse4, ...) contains that instruction. > > Is there any option in mpirun to turn on the verbosity to see more > information? > > Regards, > Mahmood > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] static linking MPI libraries with applications
Am 14.09.2016 um 20:09 schrieb Mahmood Naderan:
> I installed libibverb-devel-static.x86_64 via yum
>
>
> root@cluster:tpar# yum list libibverb*
> Installed Packages
> libibverbs.x86_64 1.1.8-4.el6
>@base
> libibverbs-devel.x86_64 1.1.8-4.el6
>@base
> libibverbs-devel-static.x86_641.1.8-4.el6
>@base
> Available Packages
> libibverbs.i686 1.1.8-4.el6
>base
> libibverbs-devel.i686 1.1.8-4.el6
>base
> libibverbs-utils.x86_64 1.1.8-4.el6
>base
> root@cluster:tpar# find /usr -name libibverb*
> /usr/lib64/libibverbs.so.1.0.0
> /usr/lib64/libibverbs.so
> /usr/lib64/libibverbs.a
> /usr/lib64/libibverbs.so.1
> /usr/share/doc/libibverbs-1.1.8
>
>
> and added /usr/lib64/libibverbs.a similar to the scalapack I added... Just
> gave the full path.
>
>
>
> However, this is what I get:
>
> libmpi_f90.a \
> `FoX/FoX-config --libs --wcml` ../libscalapack.a
> ../libopenblas.a /export/apps/siesta/openmpi-1.8.8/lib/libmpi_mpifh.a
> /export/apps/siesta/openmpi-1.8.8/lib/libmpi_usempi.a /usr/lib64/libibverbs.a
> /export/apps/siesta/openmpi-1.8.8/lib/libopen-rte.a(session_dir.o): In
> function `orte_session_dir_get_name':
> session_dir.c:(.text+0x751): warning: Using 'getpwuid' in statically linked
> applications requires at runtime the shared libraries from the glibc version
> used for linking
> sockets.o: In function `open_socket':
> sockets.c:(.text+0xb5): warning: Using 'getaddrinfo' in statically linked
> applications requires at runtime the shared libraries from the glibc version
> used for linking
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libpthread.a(libpthread.o):
> In function `sem_open':
> (.text+0x764d): warning: the use of `mktemp' is dangerous, better use
> `mkstemp'
> /export/apps/siesta/openmpi-1.8.8/lib/libopen-rte.a(ras_slurm_module.o): In
> function `init':
> ras_slurm_module.c:(.text+0x6d5): warning: Using 'gethostbyname' in
> statically linked applications requires at runtime the shared libraries from
> the glibc version used for linking
> /export/apps/siesta/openmpi-1.8.8/lib/libopen-pal.a(evutil.o): In function
> `evutil_unparse_protoname':
> /export/apps/siesta/openmpi-1.8.8/opal/mca/event/libevent2021/libevent/evutil.c:758:
> warning: Using 'getprotobynumber' in statically linked applications requires
> at runtime the shared libraries from the glibc version used for linking
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libnl.a(utils.o): In
> function `nl_str2ip_proto':
> (.text+0x599): warning: Using 'getprotobyname' in statically linked
> applications requires at runtime the shared libraries from the glibc version
> used for linking
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libibverbs.a(src_libibverbs_la-init.o):
> In function `load_driver':
> (.text+0x2ec): undefined reference to `dlopen'
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libibverbs.a(src_libibverbs_la-init.o):
> In function `load_driver':
> (.text+0x331): undefined reference to `dlerror'
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libibverbs.a(src_libibverbs_la-init.o):
> In function `ibverbs_init':
> (.text+0xd25): undefined reference to `dlopen'
> /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libibverbs.a(src_libibverbs_la-init.o):
> In function `ibverbs_init':
> (.text+0xd36): undefined reference to `dlclose'
Now you may need: -ldl
-- Reuti
> collect2: ld returned 1 exit status
> make: *** [transiesta] Error 1
>
>
>
>
> Regards,
> Mahmood
>
>
>
> On Wed, Sep 14, 2016 at 9:54 PM, Reuti wrote:
>
> The "-l" includes already the "lib" prefix when it tries to find the library.
> Hence "-libverbs" might be misleading due to the "lib" in the word, as it
> looks for "libibverbs.{a|so}". Like "-lm" will look for "libm.a" resp.
> "libm.so".
>
> -- Reuti
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] static linking MPI libraries with applications
> Am 14.09.2016 um 19:12 schrieb Mahmood Naderan :
>
> So, I used
>
> ./configure --prefix=/export/apps/siesta/openmpi-1.8.8
> --enable-mpirun-prefix-by-default --enable-static --disable-shared
> --disable-dlopen
>
> and added -static to LDFLAGS, but I get:
>
> /export/apps/siesta/openmpi-1.8.8/bin/mpifort -o transiesta -static
> libfdf.a libSiestaXC.a \
>libmpi_f90.a \
> `FoX/FoX-config --libs --wcml` ../libscalapack.a
> ../libopenblas.a /export/apps/siesta/openmpi-1.8.8/lib/libmpi_mpifh.a
> /export/apps/siesta/openmpi-1.8.8/lib/libmpi_usempi.a
> /usr/bin/ld: cannot find -libverbs
What do you have on the system for "libibverbs.*"?
> collect2: ld returned 1 exit status
>
>
> removing -static will eliminate the error but that is not what I want. Should
> I build libverbs from source first? Am I on the right direction?
The "-l" includes already the "lib" prefix when it tries to find the library.
Hence "-libverbs" might be misleading due to the "lib" in the word, as it looks
for "libibverbs.{a|so}". Like "-lm" will look for "libm.a" resp. "libm.so".
-- Reuti
> Regards,
> Mahmood
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] static linking MPI libraries with applications
> Am 14.09.2016 um 15:05 schrieb Gilles Gouaillardet > : > > in this case, you should configure OpenMPI with > --disable-shared --enable-static --disable-dlopen > > then you can manually run the mpifort link command with --showme, > so you get the fully expanded gfortran link command. > then you can edit this command, and non system libs (e.g. lapack, openmpi, > siesta) with -static, but system libs (e.g. m, ibverbs, ...) with -dynamic > you might have to manually append -ldl, though that should not be needed For me it's necessary to use -ldl, I didn't find the time to look further into it. See my post from Aug 11, 2016. With older versions of Open MPI it wasn't necessary to supply it in addition. -- Reuti > > Cheers, > > Gilles > > > > On Wednesday, September 14, 2016, Mahmood Naderan > wrote: > Well I want to omit LD_LIBRARY_PATH. For that reason I am building the binary > statically. > > > note this is not required when Open MPI is configure'd with > >--enable-mpirun-prefix-by-default > I really did that. Using Rocks-6, I installed the application and openmpi on > the shared file system (/export). > Yes it is possible to add the library paths to LD_LIBRARY_PATH, but I want to > statically put the required libraries in the binary. > > > > Regards, > Mahmood > > > > On Wed, Sep 14, 2016 at 4:44 PM, Gilles Gouaillardet > wrote: > Mahmood, > > try to prepend /export/apps/siesta/openmpi-1.8.8/lib to your $LD_LIBRARY_PATH > > note this is not required when Open MPI is configure'd with > --enable-mpirun-prefix-by-default > > > Cheers, > > Gilles > > On Wednesday, September 14, 2016, Mahmood Naderan > wrote: > Hi, > Here is the problem with statically linking an application with a program. > > by specifying the library names: > > FC=/export/apps/siesta/openmpi-1.8.8/bin/mpifort > FFLAGS=-g -Os > FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT > LDFLAGS=-static > MPI1=/export/apps/siesta/openmpi-1.8.8/lib/libmpi_mpifh.a > MPI2=/export/apps/siesta/openmpi-1.8.8/lib/libmpi_usempi.a > BLAS_LIBS=../libopenblas.a > SCALAPACK_LIBS=../libscalapack.a > LIBS=$(SCALAPACK_LIBS) $(BLAS_LIBS) $(MPI1) $(MPI2) > > > > > The output of "make" is: > > /export/apps/siesta/openmpi-1.8.8/bin/mpifort -o transiesta \ >-static automatic_cell.o > libmpi_f90.a > `FoX/FoX-config --libs --wcml` ../libscalapack.a > ../libopenblas.a /export/apps/siesta/openmpi-1.8.8/lib/libmpi_mpifh.a > /export/apps/siesta/openmpi-1.8.8/lib/libmpi_usempi.a > /export/apps/siesta/openmpi-1.8.8/lib/libopen-pal.a(dl_dlopen_module.o): In > function `dlopen_open': > dl_dlopen_module.c:(.text+0x473): warning: Using 'dlopen' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > /usr/bin/ld: cannot find -libverbs > collect2: ld returned 1 exit status > > > > > If I drop -static, the error is gone... However, ldd command shoes that > binary can not access those two MPI libraries. > > > Regards, > Mahmood > > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OS X El Capitan 10.11.6 ld: symbol(s) not found for architecture x86_64
Hi, Am 23.08.2016 um 21:43 schrieb Richard G French: > Hi, all - > I'm trying to build the SPH code Gadget2 > (http://wwwmpa.mpa-garching.mpg.de/gadget/) under OS X 10.11.6 and I am > getting the following type of error: > > 222 rfrench@cosmos> make > > mpicc main.o run.o predict.o begrun.o endrun.o global.o timestep.o init.o > restart.o io.o accel.o read_ic.o ngb.o system.o allocate.o density.o > gravtree.o hydra.o driftfac.o domain.o allvars.o potential.o forcetree.o > peano.o gravtree_forcetest.o pm_periodic.o pm_nonperiodic.o longrange.o -g > -L/opt/local/lib/mpich-mp -L/usr/local/lib -lgsl -lgslcblas -lm > -L/usr/local/lib -lrfftw_mpi -lfftw_mpi -lrfftw -lfftw-o Gadget2 By default the `mpicc` should add all necessary libraries to compile and link the application. But I wonder why you speak about mpich-mp as this is a different implementation of MPI, not Open MPI. Also the default location of Open MPI isn't mpich-mp. - what does: $ mpicc -show $ which mpicc output? - which MPI library was used to build the parallel FFTW? -- Reuti > Undefined symbols for architecture x86_64: > > "_ompi_mpi_byte", referenced from: > > _read_parameter_file in begrun.o > > _compute_global_quantities_of_system in global.o > > _restart in restart.o > > _write_file in io.o > > _read_file in read_ic.o > > _find_files in read_ic.o > > _density in density.o > > .. > > I built the mpich library using > > cd openmpi-2.0.0/ > > > ./configure > > > sudo make all install > > which installed the libraries in > > > /opt/local/lib/mpich-mp > > > > I can't seem to track down the library that contains ompi_mpi_byte. > > > > Any suggestions would be welcome. Thanks! > > Dick French > > > > > -- > Richard G. French > McDowell and Whiting Professor of Astrophysics > Chair of the Astronomy Department, Wellesley College > Director of the Whitin Observatory > Cassini Mission to Saturn Radio Science Team Leader > Wellesley, MA 02481-8203 > (781) 283-3747 > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] SGE integration broken in 2.0.0
Am 16.08.2016 um 13:26 schrieb Jeff Squyres (jsquyres): > On Aug 12, 2016, at 2:15 PM, Reuti wrote: >> >> I updated my tools to: >> >> autoconf-2.69 >> automake-1.15 >> libtool-2.4.6 >> >> but I face with Open MPI's ./autogen.pl: >> >> configure.ac:152: error: possibly undefined macro: AC_PROG_LIBTOOL >> >> I recall seeing in already before, how to get rid of it? For now I fixed the >> single source file just by hand. > > This means your Autotools installation isn't correct. A common mistake that > I've seen people do is install Autoconf, Automake, and Libtool in separate > prefixes (vs. installing all 3 into a single prefix). Thx a bunch - that was it. Despite searching for a solution I found only hints that didn't solve the issue. -- Reuti > Another common mistake is accidentally using the wrong autoconf, automake, > and/or libtool (e.g., using 2 out of the 3 from your new/correct install, but > accidentally using a system-level install for the 3rd). > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] SGE integration broken in 2.0.0
Am 12.08.2016 um 21:44 schrieb r...@open-mpi.org:
> Don’t know about the toolchain issue - I use those same versions, and don’t
> have a problem. I’m on CentOS-7, so that might be the difference?
>
> Anyway, I found the missing code to assemble the cmd line for qrsh - not sure
> how/why it got deleted.
>
> https://github.com/open-mpi/ompi/pull/1960
Yep, it's working again - thx.
But for sure there was a reason behind the removal, which may be elaborated in
the Open MPI team to avoid any side effects by fixing this issue.
-- Reuti
PS: The other items I'll investigate on Monday.
>> On Aug 12, 2016, at 12:15 PM, Reuti wrote:
>>
>>>
>>> Am 12.08.2016 um 16:52 schrieb r...@open-mpi.org:
>>>
>>> IIRC, the rationale behind adding the check was that someone using SGE
>>> wanted to specify a custom launch agent, and we were overriding it with
>>> qrsh. However, the check is incorrect as that MCA param cannot be NULL.
>>>
>>> I have updated this on master - can you see if this fixes the problem for
>>> you?
>>>
>>> https://github.com/open-mpi/ompi/pull/1957
>>
>> I updated my tools to:
>>
>> autoconf-2.69
>> automake-1.15
>> libtool-2.4.6
>>
>> but I face with Open MPI's ./autogen.pl:
>>
>> configure.ac:152: error: possibly undefined macro: AC_PROG_LIBTOOL
>>
>> I recall seeing in already before, how to get rid of it? For now I fixed the
>> single source file just by hand.
>>
>> -- Reuti
>>
>>
>>> As for the blank in the cmd line - that is likely due to a space reserved
>>> for some entry that you aren’t using (e.g., when someone manually specifies
>>> the prefix). It shouldn’t cause any harm as the cmd line parser is required
>>> to ignore spaces
>>>
>>> The -ldl problem sounds like a configuration issue - you might want to file
>>> a separate issue about it
>>>
>>>> On Aug 11, 2016, at 4:28 AM, Reuti wrote:
>>>>
>>>> Hi,
>>>>
>>>> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement,
>>>> which seems to prevent the tight integration with SGE to start:
>>>>
>>>> if (NULL == mca_plm_rsh_component.agent) {
>>>>
>>>> Why is it there (it wasn't in 1.10.3)?
>>>>
>>>> If I just remove it I get:
>>>>
>>>> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>>>> qrsh orted --hnp-topo-sig ...
>>>>
>>>> instead of the former:
>>>>
>>>> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose
>>>> orted --hnp-topo-sig ...
>>>>
>>>> So, just removing the if-statement is not a perfect cure as the
>>>> $SGE_ROOT/$ARC does not prefix `qrsh`.
>>>>
>>>> ==
>>>>
>>>> BTW: why is there blank before " orted" in the assembled command line -
>>>> and it's really in the argument when I check this on the slave nodes what
>>>> should be started by the `qrsh_starter`? As long as there is a wrapping
>>>> shell, it will be removed anyway. But in a special setup we noticed this
>>>> additional blank.
>>>>
>>>> ==
>>>>
>>>> I also notice, that I have to supply "-ldl" to `mpicc` to allow the
>>>> compilation of an application to succeed in 2.0.0.
>>>>
>>>> -- Reuti
>>>> ___
>>>> users mailing list
>>>> users@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] SGE integration broken in 2.0.0
> Am 12.08.2016 um 16:52 schrieb r...@open-mpi.org:
>
> IIRC, the rationale behind adding the check was that someone using SGE wanted
> to specify a custom launch agent, and we were overriding it with qrsh.
> However, the check is incorrect as that MCA param cannot be NULL.
>
> I have updated this on master - can you see if this fixes the problem for you?
>
> https://github.com/open-mpi/ompi/pull/1957
I updated my tools to:
autoconf-2.69
automake-1.15
libtool-2.4.6
but I face with Open MPI's ./autogen.pl:
configure.ac:152: error: possibly undefined macro: AC_PROG_LIBTOOL
I recall seeing in already before, how to get rid of it? For now I fixed the
single source file just by hand.
-- Reuti
> As for the blank in the cmd line - that is likely due to a space reserved for
> some entry that you aren’t using (e.g., when someone manually specifies the
> prefix). It shouldn’t cause any harm as the cmd line parser is required to
> ignore spaces
>
> The -ldl problem sounds like a configuration issue - you might want to file a
> separate issue about it
>
>> On Aug 11, 2016, at 4:28 AM, Reuti wrote:
>>
>> Hi,
>>
>> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, which
>> seems to prevent the tight integration with SGE to start:
>>
>> if (NULL == mca_plm_rsh_component.agent) {
>>
>> Why is it there (it wasn't in 1.10.3)?
>>
>> If I just remove it I get:
>>
>> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>> qrsh orted --hnp-topo-sig ...
>>
>> instead of the former:
>>
>> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose orted
>> --hnp-topo-sig ...
>>
>> So, just removing the if-statement is not a perfect cure as the
>> $SGE_ROOT/$ARC does not prefix `qrsh`.
>>
>> ==
>>
>> BTW: why is there blank before " orted" in the assembled command line - and
>> it's really in the argument when I check this on the slave nodes what should
>> be started by the `qrsh_starter`? As long as there is a wrapping shell, it
>> will be removed anyway. But in a special setup we noticed this additional
>> blank.
>>
>> ==
>>
>> I also notice, that I have to supply "-ldl" to `mpicc` to allow the
>> compilation of an application to succeed in 2.0.0.
>>
>> -- Reuti
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] SGE integration broken in 2.0.0
Am 12.08.2016 um 16:52 schrieb r...@open-mpi.org: > IIRC, the rationale behind adding the check was that someone using SGE wanted > to specify a custom launch agent, and we were overriding it with qrsh. > However, the check is incorrect as that MCA param cannot be NULL. > > I have updated this on master - can you see if this fixes the problem for you? > > https://github.com/open-mpi/ompi/pull/1957 As written initially, I get now this verbose output with " --mca plm_base_verbose 10": [node22:02220] mca: base: close: component isolated closed [node22:02220] mca: base: close: unloading component isolated [node22:02220] mca: base: close: component slurm closed [node22:02220] mca: base: close: unloading component slurm [node22:02220] [[28119,0],0] plm:rsh: final template argv: qrsh orted --hnp-topo-sig 2N:2S:2L3:8L2:8L1:8C:8H:x86_64 -mca ess "env" -mca ess_base_jobid "1842806784" -mca es s_base_vpid "" -mca ess_base_num_procs "9" -mca orte_hnp_uri "1842806784.0;usock;tcp://192.168.154.22,192.168.154.92:46186 " --mca plm_base_verbose "10" -mca plm "rsh" -mca pmix "^s1,s2,cray" bash: node13: command not found bash: node20: command not found bash: node12: command not found bash: node16: command not found bash: node17: command not found bash: node14: command not found bash: node15: command not found Your "qrsh" request could not be scheduled, try again later. Sure, the name of the machine is allowed only after the additional "-inherit" to `qrsh`. Please see below for the complete in 1.10.3, hence the assembly seems also not to be done in the correct way. -- Reuti > On Aug 11, 2016, at 4:28 AM, Reuti wrote: > ... > instead of the former: > > /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose orted > --hnp-topo-sig ... ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] mpirun won't find programs from the PATH environment variable that are in directories that are relative paths
Am 12.08.2016 um 20:34 schrieb r...@open-mpi.org: > Sorry for the delay - I had to catchup on some other things before I could > come back to checking this one. Took me awhile to track this down, but the > change is in test for master: > > https://github.com/open-mpi/ompi/pull/1958 > > Once complete, I’ll set it up for inclusion in v2.0.1 > > Thanks for reporting it! > Ralph > > >> On Jul 29, 2016, at 5:47 PM, Phil Regier >> wrote: >> >> If I'm reading you right, you're presently unable to do the equivalent >> (albeit probably with PATH set on a different line somewhere above) of >> >> PATH=arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana >> >> I'm mildly curious whether it would help to add a leading "./" to get the >> equivalent of >> >> PATH=./arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana >> >> But to be clear, I'm advocating >> >> PATH=$PWD/arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana >> >> as opposed to >> >> mpirun -n 1 $PWD/arch/x86_64-rhel7-gcc48-opt/bin/psana >> >> mostly because you still get to set the path once and use it many times >> without duplicating code. >> >> >> For what it's worth, I've seen Ralph's suggestion generalized to something >> like >> >> PREFIX=$PWD/arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 $PREFIX/psana AFAICS $PREFIX is evaluated too early. $ PREFIX=old_value $ PREFIX=foobar /bin/echo $PREFIX old_value Unless exactly this is the desired effect. -- Reuti >> >> where PREFIX might be set above in the same script, or sourced from a common >> config script or a custom environment module. I think this style appeals to >> many users on many levels. >> >> >> In any event, though, if this really is a bug that gets fixed, you've got >> lots of options. >> >> >> >> >> On Fri, Jul 29, 2016 at 5:24 PM, Schneider, David A. >> wrote: >> Hi, Thanks for the reply! It does look like mpirun runs from the same >> directory as where I launch it, and that the environment has the same value >> for PATH that I had before (with the relative directory in front), but of >> course, there are lots of other MPI based environment variables defined - >> maybe one of those means don't use the relative paths? >> >> Explicitly setting the path with $PWD like you say, yes, I agree that is a >> good defensive practice, but it is more cumbersome, the actually path looks >> >> mpirun -n 1 $PWD/arch/x86_64-rhel7-gcc48-opt/bin/psana >> >> best, >> >> David Schneider >> SLAC/LCLS >> >> From: users [users-boun...@lists.open-mpi.org] on behalf of Phil Regier >> [preg...@penguincomputing.com] >> Sent: Friday, July 29, 2016 5:12 PM >> To: Open MPI Users >> Subject: Re: [OMPI users] mpirun won't find programs from the PATH >> environment variable that are in directories that are relative paths >> >> I might be three steps behind you here, but does "mpirun pwd" show >> that all your launched processes are running in the same directory as the >> mpirun command? I assume that "mpirun env" would show that your PATH >> variable is being passed along correctly, since you don't have any problems >> with absolute paths. In any event, is PATH=$PWD/dir/bin not an option? >> >> Seems to me that this last would be good practice for location-sensitive >> launches in general, though I do tend to miss things. >> >> On Fri, Jul 29, 2016 at 4:34 PM, Schneider, David A. >> mailto:david...@slac.stanford.edu>> wrote: >> I am finding, on linux, rhel7, with openmpi 1.8.8 and 1.10.3, that mpirun >> won't find apps that are specified on a relative path, i.e, if I have >> >> PATH=dir/bin >> >> and I am in a directory which has dir/bin as a subdirectory, and an >> executable bir/bin/myprogram, I can't do >> >> mpirun myprogram >> >> I get the error message that >> >> mpirun was unable to find the specified executable file, and therefore >> did not launch the job. >> >> whereas if I put an absolute path, something like >> >> PATH=/home/me/dir/bin >> >> then it works. >> >> This causes some problematic silent failure, sometimes we use relative >> directories to override a 'base' release, so if I had >> >> PATH=dir/bin:/central/install/dir/bin >> >> and m
Re: [OMPI users] SGE integration broken in 2.0.0
> Am 11.08.2016 um 13:28 schrieb Reuti :
>
> Hi,
>
> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, which
> seems to prevent the tight integration with SGE to start:
>
>if (NULL == mca_plm_rsh_component.agent) {
>
> Why is it there (it wasn't in 1.10.3)?
>
> If I just remove it I get:
>
> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>qrsh orted --hnp-topo-sig ...
>
> instead of the former:
>
> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose orted
> --hnp-topo-sig ...
>
> So, just removing the if-statement is not a perfect cure as the
> $SGE_ROOT/$ARC does not prefix `qrsh`.
I forgot to mention: the original effect is, that it always tries to use `ssh`
to contact the slave nodes, despite the fact that it's running under SGE.
-- Reuti
> ==
>
> BTW: why is there blank before " orted" in the assembled command line - and
> it's really in the argument when I check this on the slave nodes what should
> be started by the `qrsh_starter`? As long as there is a wrapping shell, it
> will be removed anyway. But in a special setup we noticed this additional
> blank.
>
> ==
>
> I also notice, that I have to supply "-ldl" to `mpicc` to allow the
> compilation of an application to succeed in 2.0.0.
>
> -- Reuti
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] SGE integration broken in 2.0.0
Hi,
In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, which
seems to prevent the tight integration with SGE to start:
if (NULL == mca_plm_rsh_component.agent) {
Why is it there (it wasn't in 1.10.3)?
If I just remove it I get:
[node17:25001] [[27678,0],0] plm:rsh: final template argv:
qrsh orted --hnp-topo-sig ...
instead of the former:
/usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose orted
--hnp-topo-sig ...
So, just removing the if-statement is not a perfect cure as the $SGE_ROOT/$ARC
does not prefix `qrsh`.
==
BTW: why is there blank before " orted" in the assembled command line - and
it's really in the argument when I check this on the slave nodes what should be
started by the `qrsh_starter`? As long as there is a wrapping shell, it will be
removed anyway. But in a special setup we noticed this additional blank.
==
I also notice, that I have to supply "-ldl" to `mpicc` to allow the compilation
of an application to succeed in 2.0.0.
-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] OMPI users] OMPI users] OMPI users] OMPI users] MPI inside MPI (still)
Am 18.12.2014 um 04:24 schrieb Alex A. Schmidt :
>
> The option system("env -i ...") has been tested earlier by me and it does
> work. There is doubt though it would work along with a job scheduler.
> I will reserve this as a last resort solution.
You could also redirect the stdin before the `system()` call in your
application by dup2(). But I'm somewhat lost in the complete setup: you have a
batch scheduler, your MPI application and third party apps (using MPI, Linda,
OpenMP,...) with both of the latter working in parallel (how to forward granted
machines/cores to siesta in case it should spread further to other machines?).
What will call what? Do you submit your application or the third party apps as
mentioned earlier? Will the batch scheduler honor any redirection which you
discussed recently, and/or you can you tell the batch scheduler to use a
different stdin/-out/-err in DRMAA by setting
drmaa_input_path/drmaa_output_path/drmaa_error_path for example?
-- Reuti
> mpi_comm_spawn("/bin/sh","-c","siesta < infile",..) definitely does not work.
>
> Patching siesta to start as "siesta -in infile" sounds very promissing.
> That option as well as a "-o outfile" option should be there to start with.
> I can bring that to the attention of the siesta developers. But then we have
> this other app "dftb+", and this other one "Gaussian" (just to mention a few).
> Some of them might already act like that, others don't. So, having
> mpi_comm_spawn working with i/o redirection would be in fact more
> simpler.
>
> Perhaps a shell script "siesta_alt" could be written to convert
> "siesta_alt infile outfile" into "siesta < infile > outfile" and then
> do mpi_comm_spawn("siesta_alt","infile","outfile",...). But I am not
> sure if the spawn on siesta_alt would actually lead to a spawn on siesta
> as expected...
>
> Alex
>
> 2014-12-17 22:35 GMT-02:00 Gilles Gouaillardet
> :
> Alex,
>
> You do not want to spawn mpirun.
> Or if this is really what you want, then just use system("env -i ...")
>
> I think what you need is spawn a shell that do the redirection and then
> invoke your app.
> This is something like
> MPI_Comm_spawn("/bin/sh", "-c", "siesta < infile")
>
> That being said, i strongly recommend you patch siesta so it can be invoked
> like this
> siesta -in infile
> (plus the MPI_Comm_disconnect call explained by George)
> That would make everything so much easier
>
> Cheers,
>
> Gilles
>
> "Alex A. Schmidt" wrote:
> Let me rephrase the previous message:
>
> Putting "/bin/sh" in command with info key "ompi_non_mpi" set to ".true."
> (if command is empty, mpi_comm_spawn tries to execute ' ') of
> mpi_comm_spawn and "-c" "mpirun -n 1 myapp" in args results in
> this message:
>
> **
>
> Open MPI does not support recursive calls of mpirun
>
> **
>
> Putting a single string in args as "-c mpirun -n 1 myapp" or "-c 'mpirun -n
> 1 myapp' "
> returns
>
> /usr/bin/sh: - : invalid option
>
> Alex
>
>
>
> 2014-12-17 21:47 GMT-02:00 Alex A. Schmidt :
> Putting "/bin/sh" in command with info key "ompi_non_mpi" set to ".true."
> (if command is empty, mpi_comm_spawn tries to execute ' ') of
> mpi_comm_spawn and "-c" "mpirun -n 1 myapp" in args results in
> this message:
>
> /usr/bin/sh: -c: option requires an argument
>
> Putting a single string in args as "-c mpirun -n 1 myapp" or "-c 'mpirun -n
> 1 myapp' "
> returns
>
> /usr/bin/sh: - : invalid option
>
> Alex
>
> 2014-12-17 20:17 GMT-02:00 George Bosilca :
> I don't think this has any chance of working. The redirection is something
> interpreted by the shell, and when Open MPI "fork-exec" a process it does not
> behave as the shell.
>
> Thus a potentially non-portable solution would be to instead of launching the
> mpirun directly to launch it through a shell. Maybe something like "/bin/sh",
> "-c", "mpirun -n 1 myapp".
>
> George.
>
>
> On Wed, Dec 17, 2014 at 5:02 PM, Alex A. Schmidt wrote:
> Ralph,
>
> Sorry, "<" as an element of argv to mpi_comm_spawn is interpreted just the
> same, as another parameter by the spawnee process.
>
> But I am confuse
Re: [OMPI users] OMPI users] MPI inside MPI (still)
Hi,
Am 13.12.2014 um 02:43 schrieb Alex A. Schmidt:
> MPI_comm_disconnect seem to work but not quite.
> The call to it returns almost immediatly while
> the spawn processes keep piling up in the background
> until they are all done...
>
> I think system('env -i qsub...') to launch the third party apps
> would take the execution of every call back to the scheduler
> queue. How would I track each one for their completion?
So your goal is to implement some kind of workflow supervisor by submitting
jobs to the queuing system and act after their completion depending on the
results of the QC software? Does this workflow software need MPI to do this?
Nevertheless, submitting from an application can be done by DRMAA instead of
plain system calls. There you can also check the states of the job in the
queuing system and/or wait for them and control them more easily (i.e.
terminate, suspend,...).
-- Reuti
http://www.drmaa.org/
https://arc.liv.ac.uk/SGE/howto/howto.html#DRMAA
> Alex
>
> 2014-12-12 22:35 GMT-02:00 Gilles Gouaillardet
> :
> Alex,
>
> You need MPI_Comm_disconnect at least.
> I am not sure if this is 100% correct nor working.
>
> If you are using third party apps, why dont you do something like
> system("env -i qsub ...")
> with the right options to make qsub blocking or you manually wait for the end
> of the job ?
>
> That looks like a much cleaner and simpler approach to me.
>
> Cheers,
>
> Gilles
>
> "Alex A. Schmidt" wrote:
> Hello Gilles,
>
> Ok, I believe I have a simple toy app running as I think it should:
> 'n' parent processes running under mpi_comm_world, each one
> spawning its own 'm' child processes (each child group work
> together nicely, returning the expected result for an mpi_allreduce call).
>
> Now, as I mentioned before, the apps I want to run in the spawned
> processes are third party mpi apps and I don't think it will be possible
> to exchange messages with them from my app. So, I do I tell
> when the spawned processes have finnished running? All I have to work
> with is the intercommunicator returned from the mpi_comm_spawn call...
>
> Alex
>
>
>
>
> 2014-12-12 2:42 GMT-02:00 Alex A. Schmidt :
> Gilles,
>
> Well, yes, I guess
>
> I'll do tests with the real third party apps and let you know.
> These are huge quantum chemistry codes (dftb+, siesta and Gaussian)
> which greatly benefits from a parallel environment. My code is just
> a front end to use those, but since we have a lot of data to process
> it also benefits from a parallel environment.
>
> Alex
>
>
> 2014-12-12 2:30 GMT-02:00 Gilles Gouaillardet :
> Alex,
>
> just to make sure ...
> this is the behavior you expected, right ?
>
> Cheers,
>
> Gilles
>
>
> On 2014/12/12 13:27, Alex A. Schmidt wrote:
>> Gilles,
>>
>> Ok, very nice!
>>
>> When I excute
>>
>> do rank=1,3
>> call MPI_Comm_spawn('hello_world','
>> ',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status)
>> enddo
>>
>> I do get 15 instances of the 'hello_world' app running: 5 for each parent
>> rank 1, 2 and 3.
>>
>> Thanks a lot, Gilles.
>>
>> Best regargs,
>>
>> Alex
>>
>>
>>
>>
>> 2014-12-12 1:32 GMT-02:00 Gilles Gouaillardet >> :
>>>
>>> Alex,
>>>
>>> just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs
>>> parameter :
>>>
>>>int MPI_Comm_spawn(char *command, char *argv[], int maxprocs,
>>> MPI_Info info,
>>> int root, MPI_Comm comm, MPI_Comm *intercomm,
>>> int array_of_errcodes[])
>>>
>>> INPUT PARAMETERS
>>>maxprocs
>>> - maximum number of processes to start (integer, significant
>>> only at root)
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2014/12/12 12:23, Alex A. Schmidt wrote:
>>>
>>> Hello Gilles,
>>>
>>> Thanks for your reply. The "env -i PATH=..." stuff seems to work!!!
>>>
>>> call system("sh -c 'env -i PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2
>>> hello_world' ")
>>>
>>> did produce the expected result with a simple openmi "hello_world" code I
>>> wrote.
>>>
>>> I might be harder though with the real third party app I have in mind. And
>>> I realize
Re: [OMPI users] Cannot open configuration file - openmpi/mpic++-wrapper-data.txt
Hi, please have a look here: http://www.open-mpi.org/faq/?category=building#installdirs -- Reuti Am 09.12.2014 um 07:26 schrieb Manoj Vaghela: > Hi OpenMPI Users, > > I am trying to build OpenMPI libraries using standard configuration and > compile procedure. It is just the one thing that I want to install all in a > user specified path like following: > > OMPI_DIR is something like $HOME/Shared_Build/openmpi-1.8.3 > > [OMPI_DIR] $ ./configure --prefix=$PWD/linux_x64 > > It all went successfully and it installed all in the path above. > > I then moved the linux_x64 folder to location $HOME/mpi/openmpi/1.8.3. Now > the path of installation is $HOME/mpi/openmpi/1.8.3/linux_x64 > > I added PATH and LD_LIBRARY_PATH as below: > > export PATH=$HOME/mpi/openmpi/1.8.3/linux_x64/bin:$PATH > export LD_LIBRARY_PATH=$HOME/mpi/openmpi/1.8.3/linux_x64/lib > > which when using mpic++ command gives following: > > Cannot open configuration file > /home/manoj//linux_x64/share/openmpi/mpic++-wrapper-data.txt > Error parsing data file mpic++: Not found > > This shows the OLD installation path for which --prefix was specified. Now > the installation folder moved to NEW path. But still searches the same OLD > location. > > I searched on the web, but with that info (./configure --with-devel-headers > --enable-binaries did not work and gave the same issue) > > This question may be a repeat but please experts guide me. I also will need > to copy linux_x64 folder to other similar machine from which these libraries > can be used to compile and run application without compiling the whole source > code. > > Thanks. > > -- > regards, > Manoj > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/12/25931.php
Re: [OMPI users] How OMPI picks ethernet interfaces
Jeff, Gus, Gilles, Am 14.11.2014 um 15:56 schrieb Jeff Squyres (jsquyres): > I lurked on this thread for a while, but I have some thoughts on the many > issues that were discussed on this thread (sorry, I'm still pretty under > water trying to get ready for SC next week...). I appreciate your replies and will read them thoroughly. I think it's best to continue with the discussion after SC14. I don't want to put any burden on anyone when time is tight. -- Reuti > These points are in no particular order... > > 0. Two fundamental points have been missed in this thread: > > - A hostname technically has nothing to do with the resolvable name of an > IP interface. By convention, many people set the hostname to be the same as > some "primary" IP interface (for some definition of "primary", e.g., eth0). > But they are actually unrelated concepts. > > - Open MPI uses host specifications only to specify a remote server, *NOT* > an interface. E.g., when you list names in a hostile or the --host CLI > option, those only specify the server -- not the interface(s). This was an > intentional design choice because there tends to be confusion and different > schools of thought about the question "What's the [resolvable] name of that > remote server?" Hence, OMPI will take any old name you throw at it to > identify that remote server, but then we have separate controls for > specifying which interface(s) to use to communicate with that server. > > 1. Remember that there is at least one, and possibly two, uses of TCP > communications in Open MPI -- and they are used differently: > > - Command/control (sometimes referred to as "oob"): used for things like > mpirun control messages, shuttling IO from remote processes back to mpirun, > etc. Generally, unless you have a mountain of stdout/stderr from your > launched processes, this isn't a huge amount of traffic. > > - MPI messages: kernel-based TCP is the fallback if you don't have some > kind of faster off-server network -- i.e., the TCP BTL. Like all BTLs, the > TCP BTL carries all MPI traffic when it is used. How much traffic is > sent/received depends on your application. > > 2. For OOB, I believe that the current ORTE mechanism is that it will try all > available IP interfaces and use the *first* one that succeeds. Meaning: > after some negotiation, only one IP interface will be used to communicate > with a given peer. > > 3. The TCP BTL will examine all local IP interfaces and determine all that > can be used to reach each peer according to the algorithm described here: > http://www.open-mpi.org/faq/?category=tcp#tcp-routability-1.3. It will use > *all* IP interfaces to reach a given peer in order to maximize the available > bandwidth. > > 4. The usNIC BTL uses UDP as its wire transport, and therefore has the same > reachability issues as both the TCP OOB and BTL. However, we use a different > mechanism than the algorithm described in the above-cited FAQ item: we simply > query the Linux routing table. This can cause ARP requests, but the kernel > caches them (e.g., for multiple MPI procs on the same server making the > same/similar requests), and for a properly-segmented L3 network, each MPI > process will effectively end up querying about its local gateway (vs. the > actual peer), and therefore the chances of having that ARP already cached are > quite high. > > --> I want to make this clear: there's nothing magic our the > usNIC/check-the-routing-table approach. It's actually a very standard > IP/datacenter method. With a proper routing table, you can know fairly > quickly whether local IP interface X can reach remote IP interface Y. > > 5. The original problem cited in this thread was about the TCP OOB, not the > TCP BTL. It's important to keep straight that the OOB, with no guidance from > the user, was trying to probe the different IP interfaces and find one that > would reach a peer. Using the check-the-routing-table approach cited in #4, > we might be able to make this better (that's what Ralph and I are going to > talk about in December / post-SC / post-US Thanksgiving holiday). > > 6. As a sidenote to #5, the TCP OOB and TCP BTL determine reachability in > different ways. Remember that the TCP BTL has the benefit of having all the > ORTE infrastructure up and running. Meaning: MPI processes can exchange IP > interface information and then use that information to compute which peer IP > interfaces can be reached. The TCP OOB doesn't have this benefit -- it's > being used to establish initial connectivity. Hence, it probes each IP > interface to
Re: [OMPI users] How OMPI picks ethernet interfaces
Am 13.11.2014 um 17:14 schrieb Ralph Castain: > Hmmm…I’m beginning to grok the issue. It is a tad unusual for people to > assign different hostnames to their interfaces - I’ve seen it in the Hadoop > world, but not in HPC. Still, no law against it. Maybe it depends on the background to do it this way. At one point in the past I read this Howto: https://arc.liv.ac.uk/SGE/howto/multi_intrfcs.html and appreciated the idea to route different services to different interfaces - a large file copy won't hurt the MPI communication this way. As SGE handles it well to contact the qmaster or execds on the correct interface of the machines (which might be eth0, eth1 or any else one), I'm doing it for a decade now this way and according to the mails on the SGE lists others are doing it too. Hence I don't see it that unusual. > This will take a little thought to figure out a solution. One problem that > immediately occurs is if someone includes a hostfile that has lines which > refer to the same physical server, but using different interface names. Yes, I see this point too. Therefore I had the idea to list all the interfaces one want to use in one line. In case they put it in different lines, they would do it the wrong way - their fault. One line = one machine, unless the list of interfaces is exactly the same in multiple lines, then they could be added up like now. (Under SGE there is the [now correctly working] setup to get the same machine a couple of times in case they origin from several queues. But this would still fit with the above interpretation: the interface name is the same although they are coming from different queues they can just be added up like now in the GridEngine MCA.) > We’ll think those are completely distinct servers, and so the process > placement will be totally messed up. > > We’ll also encounter issues with the daemon when it reports back, as the > hostname it gets will almost certainly differ from the hostname we were > expecting. Not as critical, but need to check to see where that will impact > the code base Hence I prefer to use eth0 for Open MPI (for now). But I remember that there was a time when it could be set up to route the MPI traffic dedicated to eth1, although it was for MPICH(1): https://arc.liv.ac.uk/SGE/howto/mpich-integration.html => Wrong interface selected for the back channel of the MPICH-tasks with the ch_p4-device > We can look at the hostfile changes at that time - no real objection to them, > but would need to figure out how to pass that info to the appropriate > subsystems. I assume you want this to apply to both the oob and tcp/btl? Yes. > Obviously, this won’t make it for 1.8 as it is going to be fairly intrusive, > but we can probably do something for 1.9 > >> On Nov 13, 2014, at 4:23 AM, Reuti wrote: >> >> Am 13.11.2014 um 00:34 schrieb Ralph Castain: >> >>>> On Nov 12, 2014, at 2:45 PM, Reuti wrote: >>>> >>>> Am 12.11.2014 um 17:27 schrieb Reuti: >>>> >>>>> Am 11.11.2014 um 02:25 schrieb Ralph Castain: >>>>> >>>>>> Another thing you can do is (a) ensure you built with —enable-debug, and >>>>>> then (b) run it with -mca oob_base_verbose 100 (without the >>>>>> tcp_if_include option) so we can watch the connection handshake and see >>>>>> what it is doing. The —hetero-nodes will have not affect here and can be >>>>>> ignored. >>>>> >>>>> Done. It really tries to connect to the outside interface of the >>>>> headnode. But being there a firewall or not: the nodes have no clue how >>>>> to reach 137.248.0.0 - they have no gateway to this network at all. >>>> >>>> I have to revert this. They think that there is a gateway although it >>>> isn't. When I remove the entry by hand for the gateway in the routing >>>> table it starts up instantly too. >>>> >>>> While I can do this on my own cluster I still have the 30 seconds delay on >>>> a cluster where I'm not root, while this can be because of the firewall >>>> there. The gateway on this cluster is indeed going to the outside world. >>>> >>>> Personally I find this behavior a little bit too aggressive to use all >>>> interfaces. If you don't check this carefully beforehand and start a long >>>> running application one might even not notice the delay during the startup. >>> >>> Agreed - do you have any suggestions on how we should choose the order in >>> which to try them? I haven’t been able to come up with anything yet. Jeff >>> has
Re: [OMPI users] How OMPI picks ethernet interfaces
Am 13.11.2014 um 00:34 schrieb Ralph Castain: >> On Nov 12, 2014, at 2:45 PM, Reuti wrote: >> >> Am 12.11.2014 um 17:27 schrieb Reuti: >> >>> Am 11.11.2014 um 02:25 schrieb Ralph Castain: >>> >>>> Another thing you can do is (a) ensure you built with —enable-debug, and >>>> then (b) run it with -mca oob_base_verbose 100 (without the >>>> tcp_if_include option) so we can watch the connection handshake and see >>>> what it is doing. The —hetero-nodes will have not affect here and can be >>>> ignored. >>> >>> Done. It really tries to connect to the outside interface of the headnode. >>> But being there a firewall or not: the nodes have no clue how to reach >>> 137.248.0.0 - they have no gateway to this network at all. >> >> I have to revert this. They think that there is a gateway although it isn't. >> When I remove the entry by hand for the gateway in the routing table it >> starts up instantly too. >> >> While I can do this on my own cluster I still have the 30 seconds delay on a >> cluster where I'm not root, while this can be because of the firewall there. >> The gateway on this cluster is indeed going to the outside world. >> >> Personally I find this behavior a little bit too aggressive to use all >> interfaces. If you don't check this carefully beforehand and start a long >> running application one might even not notice the delay during the startup. > > Agreed - do you have any suggestions on how we should choose the order in > which to try them? I haven’t been able to come up with anything yet. Jeff has > some fancy algo in his usnic BTL that we are going to discuss after SC that > I’m hoping will help, but I’d be open to doing something better in the > interim for 1.8.4 The plain`mpiexec` should just use the specified interface it finds in the hostfile. Being it hand crafted or prepared by any queuing system. Option: could a single entry for a machine in the hostfile contain a list of interfaces? I mean something like: node01,node01-extra-eth1,node01-extra-eth2 slots=4 or node01* slots=4 Means: use exactly these interfaces or even try to find all available interfaces on/between the machines. In case all interfaces have the same name, then it's up to the admin to correct this. -- Reuti >> -- Reuti >> >> >>> It tries so independent from the internal or external name of the headnode >>> given in the machinefile - I hit ^C then. I attached the output of Open MPI >>> 1.8.1 for this setup too. >>> >>> -- Reuti >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/11/25777.php >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25781.php > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25782.php >
Re: [OMPI users] How OMPI picks ethernet interfaces
Gus, Am 13.11.2014 um 02:59 schrieb Gus Correa: > On 11/12/2014 05:45 PM, Reuti wrote: >> Am 12.11.2014 um 17:27 schrieb Reuti: >> >>> Am 11.11.2014 um 02:25 schrieb Ralph Castain: >>> >>>> Another thing you can do is (a) ensure you built with —enable-debug, >> and then (b) run it with -mca oob_base_verbose 100 >> (without the tcp_if_include option) so we can watch >> the connection handshake and see what it is doing. >> The —hetero-nodes will have not affect here and can be ignored. >>> >>> Done. It really tries to connect to the outside >> interface of the headnode. But being there a firewall or not: >> the nodes have no clue how to reach 137.248.0.0 - >> they have no gateway to this network at all. >> >> I have to revert this. >> They think that there is a gateway although it isn't. >> When I remove the entry by hand for the gateway in the >> routing table it starts up instantly too. >> >> While I can do this on my own cluster I still have the >> 30 seconds delay on a cluster where I'm not root, >> while this can be because of the firewall there. >> The gateway on this cluster is indeed going >> to the outside world. >> >> Personally I find this behavior a little bit too aggressive >> to use all interfaces. If you don't check this carefully >> beforehand and start a long running application one might >> even not notice the delay during the startup. >> >> -- Reuti >> > > Hi Reuti > > You could use the mca parameter file > (say, $prefix/etc/openmpi-mca-params.conf) to configure cluster-wide > the oob (and btl) interfaces to be used. > The users can still override your choices if they want. > > Just put a line like this in openmpi-mca-params.conf : > oob_tcp_if_include=192.168.154.0/26 > > (and similar for btl_tcp_if_include, btl_openib_if_include). > > Get a full list from "ompi_info --all --all |grep if_include". > > See these FAQ: > > http://www.open-mpi.org/faq/?category=tcp#tcp-selection > http://www.open-mpi.org/faq/?category=tuning#setting-mca-params > > Compute nodes tend to be multi-homed, so what criterion would OMPI use > to select one interface among many, My compute nodes are having two interfaces: one for MPI (and the low ssh/SGE traffic to start processes somewhere) and one for NFS to transfer files from/to the file server. So: Open MPI may use both interfaces without telling me anything about it? How will it split the traffic? 50%/50%? When there is a heavy file transfer on the NFS interface: might it hurt Open MPI's communication or will it balance the usage on-the-fly? When I prepare a machinefile with the name of the interfaces (or get the names from SGE's PE_HOSTFILE) it should use just this (except native IB), and not looking around for other paths to the other machine(s) (IMO). Therefore different interfaces have different names in my setup. "node01" is just eth0 and different from "node01-nfs" for eth1. > not knowing beforehand what exists in a particular computer? > There would be a risk to make a bad choice. > The current approach gives you everything, and you > pick/select/restrict what you want to fit your needs, > with mca parameters (which can be set in several > ways and with various scopes). > > I don't think this bad. > However, I am biased about this. > I like and use the openmpi-mca-params.conf file > to setup sensible defaults. > At least I think they are sensible. :) I see that this can be prepared for all users this way. Whenever they use my installed version it will work - maybe I have to investigate on some other clusters where I'm not root what to enter there, but it can be done for sure. BUT: it may be a rare situation that a group for quantum chemistry is having a sysadmin on their own taking care of the clusters and the well behaving operation of the installed software, being it applications or libraries. Often any PhD student in other groups will get a side project: please install software XY for the group. They are chemists and want to get the software running - they are no experts of Open MPI*. They don't care for a tight integration or using the correct interfaces as long as the application delivers the results in the end. For example: ORCA**. It's necessary for the users of the software to install a shared library of Open MPI in a specific version. I see in the ORCA*** forum that many struggle with it to compile a shared library version of Open MPI and have access to it during execution, i.e. how to set LD_LIBRARY_PATH that it's known on the slaves. The cluster admins are in another department and refuse to make any
Re: [OMPI users] OMPI users] How OMPI picks ethernet interfaces
Am 13.11.2014 um 00:55 schrieb Gilles Gouaillardet: > Could you please send the output of netstat -nr on both head and compute node > ? Head node: annemarie:~ # netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 137.248.x.y 0.0.0.0 UG0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 137.248.x.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 192.168.151.80 0.0.0.0 255.255.255.255 UH0 0 0 eth1 192.168.154.0 0.0.0.0 255.255.255.192 U 0 0 0 eth1 192.168.154.128 0.0.0.0 255.255.255.192 U 0 0 0 eth3 Compute node with (wrong) entry for the non-existing GW: node28:~ # netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.154.60 0.0.0.0 UG0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 192.168.154.0 0.0.0.0 255.255.255.192 U 0 0 0 eth0 192.168.154.64 0.0.0.0 255.255.255.192 U 0 0 0 eth1 As said: when I remove the "default" entry for the GW it starts up instantly. -- Reti > no problem obfuscating the ip of the head node, i am only interested in > netmasks and routes. > > Ralph Castain wrote: >> >>> On Nov 12, 2014, at 2:45 PM, Reuti wrote: >>> >>> Am 12.11.2014 um 17:27 schrieb Reuti: >>> >>>> Am 11.11.2014 um 02:25 schrieb Ralph Castain: >>>> >>>>> Another thing you can do is (a) ensure you built with —enable-debug, and >>>>> then (b) run it with -mca oob_base_verbose 100 (without the >>>>> tcp_if_include option) so we can watch the connection handshake and see >>>>> what it is doing. The —hetero-nodes will have not affect here and can be >>>>> ignored. >>>> >>>> Done. It really tries to connect to the outside interface of the headnode. >>>> But being there a firewall or not: the nodes have no clue how to reach >>>> 137.248.0.0 - they have no gateway to this network at all. >>> >>> I have to revert this. They think that there is a gateway although it >>> isn't. When I remove the entry by hand for the gateway in the routing table >>> it starts up instantly too. >>> >>> While I can do this on my own cluster I still have the 30 seconds delay on >>> a cluster where I'm not root, while this can be because of the firewall >>> there. The gateway on this cluster is indeed going to the outside world. >>> >>> Personally I find this behavior a little bit too aggressive to use all >>> interfaces. If you don't check this carefully beforehand and start a long >>> running application one might even not notice the delay during the startup. >> >> Agreed - do you have any suggestions on how we should choose the order in >> which to try them? I haven’t been able to come up with anything yet. Jeff >> has some fancy algo in his usnic BTL that we are going to discuss after SC >> that I’m hoping will help, but I’d be open to doing something better in the >> interim for 1.8.4 >> >>> >>> -- Reuti >>> >>> >>>> It tries so independent from the internal or external name of the headnode >>>> given in the machinefile - I hit ^C then. I attached the output of Open >>>> MPI 1.8.1 for this setup too. >>>> >>>> -- Reuti >>>> >>>> ___ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/11/25777.php >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/11/25781.php >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25782.php > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25783.php >
Re: [OMPI users] How OMPI picks ethernet interfaces
Am 12.11.2014 um 17:27 schrieb Reuti: > Am 11.11.2014 um 02:25 schrieb Ralph Castain: > >> Another thing you can do is (a) ensure you built with —enable-debug, and >> then (b) run it with -mca oob_base_verbose 100 (without the tcp_if_include >> option) so we can watch the connection handshake and see what it is doing. >> The —hetero-nodes will have not affect here and can be ignored. > > Done. It really tries to connect to the outside interface of the headnode. > But being there a firewall or not: the nodes have no clue how to reach > 137.248.0.0 - they have no gateway to this network at all. I have to revert this. They think that there is a gateway although it isn't. When I remove the entry by hand for the gateway in the routing table it starts up instantly too. While I can do this on my own cluster I still have the 30 seconds delay on a cluster where I'm not root, while this can be because of the firewall there. The gateway on this cluster is indeed going to the outside world. Personally I find this behavior a little bit too aggressive to use all interfaces. If you don't check this carefully beforehand and start a long running application one might even not notice the delay during the startup. -- Reuti > It tries so independent from the internal or external name of the headnode > given in the machinefile - I hit ^C then. I attached the output of Open MPI > 1.8.1 for this setup too. > > -- Reuti > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25777.php
Re: [OMPI users] How OMPI picks ethernet interfaces
Am 11.11.2014 um 02:25 schrieb Ralph Castain: > Another thing you can do is (a) ensure you built with —enable-debug, and then > (b) run it with -mca oob_base_verbose 100 (without the tcp_if_include > option) so we can watch the connection handshake and see what it is doing. > The —hetero-nodes will have not affect here and can be ignored. Done. It really tries to connect to the outside interface of the headnode. But being there a firewall or not: the nodes have no clue how to reach 137.248.0.0 - they have no gateway to this network at all. It tries so independent from the internal or external name of the headnode given in the machinefile - I hit ^C then. I attached the output of Open MPI 1.8.1 for this setup too. -- Reuti Wed Nov 12 16:43:12 CET 2014 [annemarie:01246] mca: base: components_register: registering oob components [annemarie:01246] mca: base: components_register: found loaded component tcp [annemarie:01246] mca: base: components_register: component tcp register function successful [annemarie:01246] mca: base: components_open: opening oob components [annemarie:01246] mca: base: components_open: found loaded component tcp [annemarie:01246] mca: base: components_open: component tcp open function successful [annemarie:01246] mca:oob:select: checking available component tcp [annemarie:01246] mca:oob:select: Querying component [tcp] [annemarie:01246] oob:tcp: component_available called [annemarie:01246] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [annemarie:01246] [[37241,0],0] oob:tcp:init rejecting loopback interface lo [annemarie:01246] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4 [annemarie:01246] [[37241,0],0] oob:tcp:init adding 137.248.x.y to our list of V4 connections [annemarie:01246] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4 [annemarie:01246] [[37241,0],0] oob:tcp:init adding 192.168.154.30 to our list of V4 connections [annemarie:01246] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 [annemarie:01246] [[37241,0],0] oob:tcp:init adding 192.168.154.187 to our list of V4 connections [annemarie:01246] [[37241,0],0] TCP STARTUP [annemarie:01246] [[37241,0],0] attempting to bind to IPv4 port 0 [annemarie:01246] [[37241,0],0] assigned IPv4 port 53661 [annemarie:01246] mca:oob:select: Adding component to end [annemarie:01246] mca:oob:select: Found 1 active transports [node28:05663] mca: base: components_register: registering oob components [node28:05663] mca: base: components_register: found loaded component tcp [node28:05663] mca: base: components_register: component tcp register function successful [node28:05663] mca: base: components_open: opening oob components [node28:05663] mca: base: components_open: found loaded component tcp [node28:05663] mca: base: components_open: component tcp open function successful [node28:05663] mca:oob:select: checking available component tcp [node28:05663] mca:oob:select: Querying component [tcp] [node28:05663] oob:tcp: component_available called [node28:05663] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [node28:05663] [[37241,0],1] oob:tcp:init rejecting loopback interface lo [node28:05663] WORKING INTERFACE 2 KERNEL INDEX 2 FAMILY: V4 [node28:05663] [[37241,0],1] oob:tcp:init adding 192.168.154.28 to our list of V4 connections [node28:05663] WORKING INTERFACE 3 KERNEL INDEX 3 FAMILY: V4 [node28:05663] [[37241,0],1] oob:tcp:init adding 192.168.154.98 to our list of V4 connections [node28:05663] [[37241,0],1] TCP STARTUP [node28:05663] [[37241,0],1] attempting to bind to IPv4 port 0 [node28:05663] [[37241,0],1] assigned IPv4 port 45802 [node28:05663] mca:oob:select: Adding component to end [node28:05663] mca:oob:select: Found 1 active transports [node28:05663] [[37241,0],1]: set_addr to uri 2440626176.0;tcp://137.248.x.y,192.168.154.30,192.168.154.187:53661 [node28:05663] [[37241,0],1]:set_addr checking if peer [[37241,0],0] is reachable via component tcp [node28:05663] [[37241,0],1] oob:tcp: working peer [[37241,0],0] address tcp://137.248.x.y,192.168.154.30,192.168.154.187:53661 [node28:05663] [[37241,0],1] PASSING ADDR 137.248.x.y TO MODULE [node28:05663] [[37241,0],1]:tcp set addr for peer [[37241,0],0] [node28:05663] [[37241,0],1] PASSING ADDR 192.168.154.30 TO MODULE [node28:05663] [[37241,0],1]:tcp set addr for peer [[37241,0],0] [node28:05663] [[37241,0],1] PASSING ADDR 192.168.154.187 TO MODULE [node28:05663] [[37241,0],1]:tcp set addr for peer [[37241,0],0] [node28:05663] [[37241,0],1]: peer [[37241,0],0] is reachable via component tcp [node28:05663] [[37241,0],1] OOB_SEND: rml_oob_send.c:199 [node28:05663] [[37241,0],1]:tcp:processing set_peer cmd [node28:05663] [[37241,0],1] SET_PEER ADDING PEER [[37241,0],0] [node28:05663] [[37241,0],1] set_peer: peer [[37241,0],0] is listening on net 137.248.x.y port 53661 [node28:05663] [[37241,0],1]:tcp:processing set_peer cmd [node28:05663] [[37241,0],1] set_peer: peer [[37241,0],0] is listening on net 192.168.154.30 port 53661 [node28:05663] [[37241,0],1]:tc
Re: [OMPI users] How OMPI picks ethernet interfaces
Am 11.11.2014 um 02:12 schrieb Gilles Gouaillardet: > Hi, > > IIRC there were some bug fixes between 1.8.1 and 1.8.2 in order to really use > all the published interfaces. > > by any change, are you running a firewall on your head node ? Yes, but only for the interface to the outside world. Nevertheless I switched it off and the result was the same 2 minutes delay during startup. > one possible explanation is the compute node tries to access the public > interface of the head node, and packets get dropped by the firewall. > > if you are running a firewall, can you make a test without it ? > /* if you do need NAT, then just remove the DROP and REJECT rules "/ > > an other possible explanation is the compute node is doing (reverse) dns > requests with the public name and/or ip of the head node and that takes some > time to complete (success or failure, this does not really matter here) I tried in the machinefile the internal and the external name of the headnode, i.e. different names for different interfaces. The result is the same. > /* a simple test is to make sure all the hosts/ip of the head node are in the > /etc/hosts of the compute node */ > > could you check your network config (firewall and dns) ? > > can you reproduce the delay when running mpirun on the head node and with one > mpi task on the compute node ? You mean one on the head node and one on the compute node, opposed to two + two in my initial test? Sure, but with 1+1 I get the same result. > if yes, then the hard way to trace the delay issue would be to strace -ttt > both orted and mpi task that are launched on the compute node and see where > the time is lost. > /* at this stage, i would suspect orted ... */ As the `ssh` on the headnode hangs for a while, I suspect it's something on the compute node. I see there during the startup: orted -mca ess env -mca orte_ess_jobid 2412773376 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2 -mca orte_hnp_uri 2412773376.0;tcp://137.248.x.y,192.168.154.30,192.168.154.187:58782 --tree-spawn -mca plm rsh === Only the subnet 192.168.154.0/26 (yes, 26) is used to access the nodes from the master i.e. login machine. As an additional information: the nodes have two network interfaces: one in 192.168.154.0/26 and one in 192.168.154.64/26 to reach a file server. == Falling back to 1.8.1 I see: bash -c orted -mca ess env -mca orte_ess_jobid 3182034944 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2 -mca orte_hnp_uri "3182034944.0;tcp://137.248.x.y,192.168.154.30,192.168.154.187:54436" --tree-spawn -mca plm rsh -mca hwloc_base_binding_policy none So, the bash was removed. But I don't think that this causes anything. -- Reuti > Cheers, > > Gilles > > On Mon, Nov 10, 2014 at 5:56 PM, Reuti wrote: > Hi, > > Am 10.11.2014 um 16:39 schrieb Ralph Castain: > > > That is indeed bizarre - we haven’t heard of anything similar from other > > users. What is your network configuration? If you use oob_tcp_if_include or > > exclude, can you resolve the problem? > > Thx - this option helped to get it working. > > These tests were made for sake of simplicity between the headnode of the > cluster and one (idle) compute node. I tried then between the (identical) > compute nodes and this worked fine. The headnode of the cluster and the > compute node are slightly different though (i.e. number of cores), and using > eth1 resp. eth0 for the internal network of the cluster. > > I tried --hetero-nodes with no change. > > Then I turned to: > > reuti@annemarie:~> date; mpiexec -mca btl self,tcp --mca oob_tcp_if_include > 192.168.154.0/26 -n 4 --hetero-nodes --hostfile machines ./mpihello; date > > and the application started instantly. On another cluster, where the headnode > is identical to the compute nodes but with the same network setup as above, I > observed a delay of "only" 30 seconds. Nevertheless, also on this cluster the > working addition was the correct "oob_tcp_if_include" to solve the issue. > > The questions which remain: a) is this a targeted behavior, b) what changed > in this scope between 1.8.1 and 1.8.2? > > -- Reuti > > > > > >> On Nov 10, 2014, at 4:50 AM, Reuti wrote: > >> > >> Am 10.11.2014 um 12:50 schrieb Jeff Squyres (jsquyres): > >> > >>> Wow, that's pretty terrible! :( > >>> > >>> Is the behavior BTL-specific, perchance? E.G., if you only use certain > >>> BTLs, does the delay disappear? > >> > >> You mean something like: > >> > >> reuti@annemarie:~> date; mpiexec -mca btl self,tcp -n 4 --hostfile > >> machines ./mpihello; date > >> M
Re: [OMPI users] oversubscription of slots with GridEngine
Am 11.11.2014 um 19:29 schrieb Ralph Castain: > >> On Nov 11, 2014, at 10:06 AM, Reuti wrote: >> >> Am 11.11.2014 um 17:52 schrieb Ralph Castain: >> >>> >>>> On Nov 11, 2014, at 7:57 AM, Reuti wrote: >>>> >>>> Am 11.11.2014 um 16:13 schrieb Ralph Castain: >>>> >>>>> This clearly displays the problem - if you look at the reported >>>>> “allocated nodes”, you see that we only got one node (cn6050). This is >>>>> why we mapped all your procs onto that node. >>>>> >>>>> So the real question is - why? Can you show us the content of PE_HOSTFILE? >>>>> >>>>> >>>>>> On Nov 11, 2014, at 4:51 AM, SLIM H.A. wrote: >>>>>> >>>>>> Dear Reuti and Ralph >>>>>> >>>>>> Below is the output of the run for openmpi 1.8.3 with this line >>>>>> >>>>>> mpirun -np $NSLOTS --display-map --display-allocation --cpus-per-proc 1 >>>>>> $exe >>>>>> >>>>>> >>>>>> master=cn6050 >>>>>> PE=orte >>>>>> JOB_ID=2482923 >>>>>> Got 32 slots. >>>>>> slots: >>>>>> cn6050 16 par6.q@cn6050 >>>>>> cn6045 16 par6.q@cn6045 >>>> >>>> The above looks like the PE_HOSTFILE. So it should be 16 slots per node. >>> >>> Hey Reuti >>> >>> Is that the standard PE_HOSTFILE format? I’m looking at the ras/gridengine >>> module, and it looks like it is expecting a different format. I suspect >>> that is the problem >> >> Well, the fourth column can be a processer range in older versions of SGE >> and the binding in newer ones, but the first three columns were always this >> way. > > Hmmm…perhaps I’m confused here. I guess you’re saying that just the last two > lines of his output contain the PE_HOSTFILE, as opposed to the entire thing? Yes. The entire thing looks like an output of the jobscript from the OP. Only the last two lines should be the content of the PE_HOSTFILE > If so, I’m wondering if that NULL he shows in there is the source of the > trouble. The parser doesn’t look like it would handle that very well, though > I’d need to test it. Is that NULL expected? Or is the NULL not really in the > file? I must admit here: for me the fourth column is either literally UNDEFINED or the tuple cpu,core in case of turned on binding like 0,0 But it's never , neither literally nor the byte 0x00. Maybe the OP can tell us which GE version he uses,. -- Reuti >> -- Reuti >> >> >>> Ralph >>> >>>> >>>> I wonder whether any environment variable was reset, which normally allows >>>> Open MPI to discover that it's running inside SGE. >>>> >>>> I.e. SGE_ROOT, JOB_ID, ARC and PE_HOSTFILE are untouched before the job >>>> starts? >>>> >>>> Supplying "-np $NSLOTS" shouldn't be necessary though. >>>> >>>> -- Reuti >>>> >>>> >>>> >>>>>> Tue Nov 11 12:37:37 GMT 2014 >>>>>> >>>>>> == ALLOCATED NODES == >>>>>> cn6050: slots=16 max_slots=0 slots_inuse=0 state=UP >>>>>> = >>>>>> Data for JOB [57374,1] offset 0 >>>>>> >>>>>> JOB MAP >>>>>> >>>>>> Data for node: cn6050 Num slots: 16 Max slots: 0Num procs: 32 >>>>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 0 >>>>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 1 >>>>>> >>>>>> … >>>>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 31 >>>>>> >>>>>> >>>>>> Also >>>>>> ompi_info | grep grid >>>>>> gives MCA ras: gridengine (MCA v2.0, API v2.0, Component >>>>>> v1.8.3) >>>>>> and >>>>>> ompi_info | grep psm >>>>>> gives MCA mtl: psm (MCA v2.0, API v2.0, Component v1.8.3) >>>>>> because the intercoonect is TrueScale/QLogic >>>>>> >>>>>> and >>
Re: [OMPI users] oversubscription of slots with GridEngine
Am 11.11.2014 um 17:52 schrieb Ralph Castain: > >> On Nov 11, 2014, at 7:57 AM, Reuti wrote: >> >> Am 11.11.2014 um 16:13 schrieb Ralph Castain: >> >>> This clearly displays the problem - if you look at the reported “allocated >>> nodes”, you see that we only got one node (cn6050). This is why we mapped >>> all your procs onto that node. >>> >>> So the real question is - why? Can you show us the content of PE_HOSTFILE? >>> >>> >>>> On Nov 11, 2014, at 4:51 AM, SLIM H.A. wrote: >>>> >>>> Dear Reuti and Ralph >>>> >>>> Below is the output of the run for openmpi 1.8.3 with this line >>>> >>>> mpirun -np $NSLOTS --display-map --display-allocation --cpus-per-proc 1 >>>> $exe >>>> >>>> >>>> master=cn6050 >>>> PE=orte >>>> JOB_ID=2482923 >>>> Got 32 slots. >>>> slots: >>>> cn6050 16 par6.q@cn6050 >>>> cn6045 16 par6.q@cn6045 >> >> The above looks like the PE_HOSTFILE. So it should be 16 slots per node. > > Hey Reuti > > Is that the standard PE_HOSTFILE format? I’m looking at the ras/gridengine > module, and it looks like it is expecting a different format. I suspect that > is the problem Well, the fourth column can be a processer range in older versions of SGE and the binding in newer ones, but the first three columns were always this way. -- Reuti > Ralph > >> >> I wonder whether any environment variable was reset, which normally allows >> Open MPI to discover that it's running inside SGE. >> >> I.e. SGE_ROOT, JOB_ID, ARC and PE_HOSTFILE are untouched before the job >> starts? >> >> Supplying "-np $NSLOTS" shouldn't be necessary though. >> >> -- Reuti >> >> >> >>>> Tue Nov 11 12:37:37 GMT 2014 >>>> >>>> == ALLOCATED NODES == >>>> cn6050: slots=16 max_slots=0 slots_inuse=0 state=UP >>>> = >>>> Data for JOB [57374,1] offset 0 >>>> >>>> JOB MAP >>>> >>>> Data for node: cn6050 Num slots: 16 Max slots: 0Num procs: 32 >>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 0 >>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 1 >>>> >>>> … >>>> Process OMPI jobid: [57374,1] App: 0 Process rank: 31 >>>> >>>> >>>> Also >>>> ompi_info | grep grid >>>> gives MCA ras: gridengine (MCA v2.0, API v2.0, Component >>>> v1.8.3) >>>> and >>>> ompi_info | grep psm >>>> gives MCA mtl: psm (MCA v2.0, API v2.0, Component v1.8.3) >>>> because the intercoonect is TrueScale/QLogic >>>> >>>> and >>>> >>>> setenv OMPI_MCA_mtl "psm" >>>> >>>> is set in the script. This is the PE >>>> >>>> pe_name orte >>>> slots 4000 >>>> user_listsNONE >>>> xuser_lists NONE >>>> start_proc_args /bin/true >>>> stop_proc_args/bin/true >>>> allocation_rule $fill_up >>>> control_slavesTRUE >>>> job_is_first_task FALSE >>>> urgency_slots min >>>> >>>> Many thanks >>>> >>>> Henk >>>> >>>> >>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain >>>> Sent: 10 November 2014 05:07 >>>> To: Open MPI Users >>>> Subject: Re: [OMPI users] oversubscription of slots with GridEngine >>>> >>>> You might also add the —display-allocation flag to mpirun so we can see >>>> what it thinks the allocation looks like. If there are only 16 slots on >>>> the node, it seems odd that OMPI would assign 32 procs to it unless it >>>> thinks there is only 1 node in the job, and oversubscription is allowed >>>> (which it won’t be by default if it read the GE allocation) >>>> >>>> >>>> On Nov 9, 2014, at 9:56 AM, Reuti wrote: >>>> >>>> Hi, >>>> >>>> >>>> Am 09.11.2014 um 18:20 schrieb SLIM H.A. : >>&
Re: [OMPI users] oversubscription of slots with GridEngine
Am 11.11.2014 um 16:13 schrieb Ralph Castain: > This clearly displays the problem - if you look at the reported “allocated > nodes”, you see that we only got one node (cn6050). This is why we mapped all > your procs onto that node. > > So the real question is - why? Can you show us the content of PE_HOSTFILE? > > >> On Nov 11, 2014, at 4:51 AM, SLIM H.A. wrote: >> >> Dear Reuti and Ralph >> >> Below is the output of the run for openmpi 1.8.3 with this line >> >> mpirun -np $NSLOTS --display-map --display-allocation --cpus-per-proc 1 $exe >> >> >> master=cn6050 >> PE=orte >> JOB_ID=2482923 >> Got 32 slots. >> slots: >> cn6050 16 par6.q@cn6050 >> cn6045 16 par6.q@cn6045 The above looks like the PE_HOSTFILE. So it should be 16 slots per node. I wonder whether any environment variable was reset, which normally allows Open MPI to discover that it's running inside SGE. I.e. SGE_ROOT, JOB_ID, ARC and PE_HOSTFILE are untouched before the job starts? Supplying "-np $NSLOTS" shouldn't be necessary though. -- Reuti >> Tue Nov 11 12:37:37 GMT 2014 >> >> == ALLOCATED NODES == >> cn6050: slots=16 max_slots=0 slots_inuse=0 state=UP >> = >> Data for JOB [57374,1] offset 0 >> >> JOB MAP >> >> Data for node: cn6050 Num slots: 16 Max slots: 0Num procs: 32 >> Process OMPI jobid: [57374,1] App: 0 Process rank: 0 >> Process OMPI jobid: [57374,1] App: 0 Process rank: 1 >> >> … >> Process OMPI jobid: [57374,1] App: 0 Process rank: 31 >> >> >> Also >> ompi_info | grep grid >> gives MCA ras: gridengine (MCA v2.0, API v2.0, Component >> v1.8.3) >> and >> ompi_info | grep psm >> gives MCA mtl: psm (MCA v2.0, API v2.0, Component v1.8.3) >> because the intercoonect is TrueScale/QLogic >> >> and >> >> setenv OMPI_MCA_mtl "psm" >> >> is set in the script. This is the PE >> >> pe_name orte >> slots 4000 >> user_listsNONE >> xuser_lists NONE >> start_proc_args /bin/true >> stop_proc_args/bin/true >> allocation_rule $fill_up >> control_slavesTRUE >> job_is_first_task FALSE >> urgency_slots min >> >> Many thanks >> >> Henk >> >> >> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain >> Sent: 10 November 2014 05:07 >> To: Open MPI Users >> Subject: Re: [OMPI users] oversubscription of slots with GridEngine >> >> You might also add the —display-allocation flag to mpirun so we can see what >> it thinks the allocation looks like. If there are only 16 slots on the node, >> it seems odd that OMPI would assign 32 procs to it unless it thinks there is >> only 1 node in the job, and oversubscription is allowed (which it won’t be >> by default if it read the GE allocation) >> >> >> On Nov 9, 2014, at 9:56 AM, Reuti wrote: >> >> Hi, >> >> >> Am 09.11.2014 um 18:20 schrieb SLIM H.A. : >> >> We switched on hyper threading on our cluster with two eight core sockets >> per node (32 threads per node). >> >> We configured gridengine with 16 slots per node to allow the 16 extra >> threads for kernel process use but this apparently does not work. Printout >> of the gridengine hostfile shows that for a 32 slots job, 16 slots are >> placed on each of two nodes as expected. Including the openmpi --display-map >> option shows that all 32 processes are incorrectly placed on the head node. >> >> You mean the master node of the parallel job I assume. >> >> >> Here is part of the output >> >> master=cn6083 >> PE=orte >> >> What allocation rule was defined for this PE - "control_slave yes" is set? >> >> >> JOB_ID=2481793 >> Got 32 slots. >> slots: >> cn6083 16 par6.q@cn6083 >> cn6085 16 par6.q@cn6085 >> Sun Nov 9 16:50:59 GMT 2014 >> Data for JOB [44767,1] offset 0 >> >> JOB MAP >> >> Data for node: cn6083 Num slots: 16 Max slots: 0Num procs: 32 >> Process OMPI jobid: [44767,1] App: 0 Process rank: 0 >> Process OMPI jobid: [44767,1] App: 0 Process rank:
