Hello,
Jeff, your fix brakes my system again. Actually you just reverted my
changes. Here is what I have:
configure:5441: *** GNU libltdl setup
configure:296939: checking location of libltdl
configure:296952: result: internal copy
configure:297028: OPAL configuring in opal/libltdl
configure:297113: running /bin/bash '.../opal/libltdl/configure'
'--prefix=.../ompi-pmix-refactoring_install/' '--enable-debug'
'--disable-oshmem' '--with-pmi=/home/artpol/sandboxes/slurm/'
--enable-ltdl-convenience --disable-ltdl-install --enable-shared
--disable-static --cache-file=/dev/null --srcdir=.../opal/libltdl
--disable-option-checking
configure:297119: /bin/bash '.../opal/libltdl/configure' succeeded for
opal/libltdl
In file included from conftest.c:718:0:
.../opal/libltdl/ltdl.h:36:31: fatal error: libltdl/lt_system.h: No such
file or directory
#include <libltdl/lt_system.h>
^
compilation terminated.
configure:297864: checking for lt_dladvise
configure:297870: result: no
configure:297923: creating ./config.lt
Surprisingly to me this error (I am sure!) occurs on any system but only on
mine it fails to set advise on! I checked that on other machines!
The reason was pointed in original PR:
ltdl.h has includes
#include < libltdl/lt_system.h >
#include < libltdl/lt_error.h >
That can't be found without "-I$srcdir/opal/libltdl/".
The point is that we DO need "-I$srcdir/opal/libltdl/" but we ALSO need
"-I$srcdir" too! I filed the new PR (
https://github.com/open-mpi/ompi/pull/301) but won't merge it until Edgar
confirms that it's OK with his system.
So the original error was in removing -I$srcdir. I was sure that we
converged on this and found another valuable discussion in ompi-release:
https://github.com/open-mpi/ompi-release/pull/34
There I was looking into configure script and found:
CPPFLAGS="-I$srcdir/ -I$srcdir/opal/libltdl/"# Must specifically
mention $srcdir here for VPATH builds# (this file is in the src tree).
cat confdefs.h - <<_ACEOF >conftest.$ac_ext/* end confdefs.h.
*/#include <$srcdir/opal/libltdl/ltdl.h>_ACEOF
And it was obvious that we don't need "-I$srcdir/" because it was hardcoded
in the include but it turns out that I've been wrong and maybe some other
building system emmits different code. I would like to see Edgars original
config.log. Jeff could you send it to me directly?
So, everybody, sorry for inconvinience!
2014-12-03 0:41 GMT+06:00 Jeff Squyres (jsquyres) <[email protected]>:
> See https://github.com/open-mpi/ompi/pull/298 for a fix.
>
> There's 2 commits on that PR -- the 2nd is just a cleanup. The real fix
> is the 1st commit, here:
>
>
> https://github.com/jsquyres/ompi/commit/a736d83fb9a7b27986a008a2cda6eb1fea839fb3
>
> If someone can confirm that this works for them, we can bring it to master.
>
> It may have the side effect of "fixing / working around" (by coincidence)
> the SLURM bug (we all agree that the Right solution is to have SLURM fix it
> upstream, but I think this will put us back in the case of "working by
> accident / despite the SLURM bug").
>
>
>
> On Dec 2, 2014, at 10:59 AM, Jeff Squyres (jsquyres) <[email protected]>
> wrote:
>
> > I'm able to replicate Edgar's problem.
> >
> > I'm investigating...
> >
> >
> > On Dec 2, 2014, at 10:39 AM, Edgar Gabriel <[email protected]> wrote:
> >
> >> the mailing list refused to let me add the config.log file, since it is
> too large, I can forward the output to you directly as well (as I did to
> Jeff).
> >>
> >> I honestly have not looked into the configure logic, I can just tell
> that OPAL_HAVE_LTDL_ADVISE is not set on my linux system for master, but is
> set on the 1.8 series (1.8 series checkout was from Nov. 20, so if
> something changed in between the result might be different).
> >>
> >>
> >>
> >> On 12/2/2014 9:27 AM, Artem Polyakov wrote:
> >>>
> >>> 2014-12-02 20:59 GMT+06:00 Edgar Gabriel <[email protected]
> >>> <mailto:[email protected]>>:
> >>>
> >>> didn't want to interfere with this thread, although I have a similar
> >>> issue, since I have the solution nearly fully cooked up. But anyway,
> >>> this last email gave the hint on why we have suddenly the problem in
> >>> ompio:
> >>>
> >>> it looks like OPAL_HAVE_LTDL_ADVISE (at least on my systems) is not
> >>> set anymore, so the entire section is being skipped. I double
> >>> checked that with the 1.8 branch, it goes through the section, but
> >>> not with master.
> >>>
> >>>
> >>> Hi, Edgar.
> >>>
> >>> Both master and ompi-release (isn't it 1.8?!) are equal in sence of my
> >>> fix. Something else!? I'd like to see config.log too but will look into
> >>> it only tomorrow.
> >>>
> >>> Also I want to add that SLURM PMI2 communicates with local slurmstepd's
> >>> and doesn't need any authentification. All PMI1 processes otherwise
> >>> communicate to the srun process and thus need libslurm services for
> >>> communication and authentification.
> >>>
> >>>
> >>> Thanks
> >>> Edgar
> >>>
> >>>
> >>>
> >>>
> >>> On 12/2/2014 7:56 AM, Jeff Squyres (jsquyres) wrote:
> >>>
> >>> Looks like I was totally lying in
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16381.php
> >>> <http://www.open-mpi.org/community/lists/devel/2014/12/16381.php>
> (where
> >>> I said we should not use RTLD_GLOBAL). We *do* use RTLD_GLOBAL:
> >>>
> >>>
> https://github.com/open-mpi/__ompi/blob/master/opal/mca/__base/mca_base_component___repository.c#L124
> >>> <
> https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_component_repository.c#L124
> >
> >>>
> >>> This ltdl advice object is passed to lt_dlopen() for all
> >>> components. My mistake; sorry.
> >>>
> >>> So the idea that using RTLD_GLOBAL will fix this SLURM bug is
> >>> incorrect.
> >>>
> >>> I believe someone said earlier in the thread that adding the
> >>> right -llibs to the configure line will solve the issue, and
> >>> that sounds correct to me. If there's a missing symbol because
> >>> the SLURM libraries are not automatically pulling in the right
> >>> dependent libraries, then *if* we put a workaround in OMPI to
> >>> fix this issue, then the right workaround is to add the relevant
> >>> -llibs when that component is linked.
> >>>
> >>> *If* you add that workaround (which is a whole separate
> >>> discussion), I would suggest adding a configure.m4 test to see
> >>> if adding the additional -llibs are necessary. Perhaps
> >>> AC_LINK_IFELSE looking for a symbol, and then if that fails,
> >>> AC_LINK_IFELSE again with the additional -llibs to see if that
> >>> works.
> >>>
> >>> Or something like that.
> >>>
> >>>
> >>>
> >>> On Dec 2, 2014, at 6:38 AM, Artem Polyakov <[email protected]
> >>> <mailto:[email protected]>> wrote:
> >>>
> >>> Agree. First you should check is to what value
> >>> OPAL_HAVE_LTDL_ADVISE is set. If it is zero - very probably
> >>> this is the same bug as mine.
> >>>
> >>> 2014-12-02 17:33 GMT+06:00 Ralph Castain <[email protected]
> >>> <mailto:[email protected]>>:
> >>> It does look similar - question is: why didn’t this fix the
> >>> problem? Will have to investigate.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Dec 2, 2014, at 3:17 AM, Artem Polyakov
> >>> <[email protected] <mailto:[email protected]>> wrote:
> >>>
> >>>
> >>>
> >>> 2014-12-02 17:13 GMT+06:00 Ralph Castain
> >>> <[email protected] <mailto:[email protected]>>:
> >>> Hmmm…if that is true, then it didn’t fix this problem as
> >>> it is being reported in the master.
> >>>
> >>> I had this problem on my laptop installation. You can
> >>> check my report it was detailed enough and see if you
> >>> hitting the same issue. My fix was also included into
> >>> 1.8 branch. I am not sure that this is the same issue
> >>> but they looks similar.
> >>>
> >>>
> >>>
> >>> On Dec 1, 2014, at 9:40 PM, Artem Polyakov
> >>> <[email protected] <mailto:[email protected]>>
> wrote:
> >>>
> >>> I think this might be related to the configuration
> >>> problem I was fixing with Jeff few months ago. Refer
> >>> here:
> >>> https://github.com/open-mpi/__ompi/pull/240
> >>> <https://github.com/open-mpi/ompi/pull/240>
> >>>
> >>> 2014-12-02 10:15 GMT+06:00 Ralph Castain
> >>> <[email protected] <mailto:[email protected]>>:
> >>> If it isn’t too much trouble, it would be good to
> >>> confirm that it remains broken. I strongly suspect
> >>> it is based on Moe’s comments.
> >>>
> >>> Obviously, other people are making this work. For
> >>> Intel MPI, all you do is point it at libpmi and they
> >>> can run. However, they do explicitly dlopen it in
> >>> their code, and I don’t know what flags they might
> >>> pass when they do so.
> >>>
> >>> If necessary, I suppose we could follow that
> >>> pattern. In other words, rather than specifically
> >>> linking the “s1” component to libpmi, instead
> >>> require that the user point us to a pmi library via
> >>> an MCA param, then explicitly dlopen that library
> >>> with RTLD_GLOBAL. This avoids the issues cited by
> >>> Jeff, but resolves the pmi linkage problem.
> >>>
> >>>
> >>> On Dec 1, 2014, at 8:09 PM, Gilles Gouaillardet
> >>> <[email protected]
> >>> <mailto:[email protected]>__> wrote:
> >>>
> >>> $ srun --version
> >>> slurm 2.6.6-VENDOR_PROVIDED
> >>>
> >>> $ srun --mpi=pmi2 -n 1 ~/hw
> >>> I am 0 / 1
> >>>
> >>> $ srun -n 1 ~/hw
> >>> /csc/home1/gouaillardet/hw: symbol lookup error:
> >>> /usr/lib64/slurm/auth_munge.__so: undefined
> >>> symbol: slurm_verbose
> >>> srun: error: slurm_receive_msg: Zero Bytes were
> >>> transmitted or received
> >>> srun: error: slurm_receive_msg[10.0.3.15]: Zero
> >>> Bytes were transmitted or received
> >>> srun: error: soleil: task 0: Exited with exit
> >>> code 127
> >>>
> >>> $ ldd /usr/lib64/slurm/auth_munge.so
> >>> linux-vdso.so.1 => (0x00007fff54478000)
> >>> libmunge.so.2 => /usr/lib64/libmunge.so.2
> >>> (0x00007f744760f000)
> >>> libpthread.so.0 => /lib64/libpthread.so.0
> >>> (0x00007f74473f1000)
> >>> libc.so.6 => /lib64/libc.so.6
> >>> (0x00007f744705d000)
> >>> /lib64/ld-linux-x86-64.so.2
> >>> (0x0000003bf5400000)
> >>>
> >>>
> >>> now, if i reling auth_munge.so so it depends on
> >>> libslurm :
> >>>
> >>> $ srun -n 1 ~/hw
> >>> srun: symbol lookup error:
> >>> /usr/lib64/slurm/auth_munge.__so: undefined
> >>> symbol: slurm_auth_get_arg_desc
> >>>
> >>>
> >>> i can give a try to the latest slurm if needed
> >>>
> >>> Cheers,
> >>>
> >>> Gilles
> >>>
> >>>
> >>> On 2014/12/02 12:56, Ralph Castain wrote:
> >>>
> >>> Out of curiosity - how are you testing
> >>> these? I have more current versions of Slurm
> >>> and would like to test the observations
> there.
> >>>
> >>>
> >>> On Dec 1, 2014, at 7:49 PM, Gilles
> >>> Gouaillardet
> >>> <[email protected]
> >>> <mailto:[email protected]
> >__>
> >>> wrote:
> >>>
> >>> I d like to make a step back ...
> >>>
> >>> i previously tested with slurm 2.6.0,
> >>> and it complained about the
> >>> slurm_verbose symbol that is defined in
> >>> libslurm.so
> >>> so with slurm 2.6.0, RTLD_GLOBAL or
> >>> relinking is ok
> >>>
> >>> now i tested with slurm 2.6.6 and it
> >>> complains about the
> >>> slurm_auth_get_arg_desc symbol, and this
> >>> symbol is not
> >>> defined in any dynamic library. it is
> >>> internally defined in the static
> >>> libcommon.a library, which is used to
> >>> build the slurm binaries.
> >>>
> >>> as far as i understand, auth_munge.so
> >>> can only be invoked from a slurm binary,
> >>> which means it cannot be invoked from an
> >>> mpi application
> >>> even if it is linked with libslurm,
> >>> libpmi, ...
> >>>
> >>> that looks like a slurm design issue
> >>> that the slurm folks will take care of.
> >>>
> >>> Cheers,
> >>>
> >>> Gilles
> >>>
> >>> On 2014/12/02 12:33, Ralph Castain wrote:
> >>>
> >>> Another option is to simply add the
> >>> -lslurm -lauth flags to the pmix/s1
> >>> component as this is the only place
> >>> that requires it, and it won’t hurt
> >>> anything to do so.
> >>>
> >>>
> >>>
> >>> On Dec 1, 2014, at 6:03 PM,
> >>> Gilles Gouaillardet
> >>> <[email protected]
> >>> <mailto:
> [email protected]>__>
> >>> <mailto:gilles.gouaillardet@__
> iferc.org
> >>> <mailto:
> [email protected]>>
> >>> wrote:
> >>>
> >>> Jeff,
> >>>
> >>> FWIW, you can read my analysis
> >>> of what is going wrong at
> >>>
> >>>
> http://www.open-mpi.org/__community/lists/pmix-devel/__2014/11/0293.php
> >>> <
> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>
> >>> <
> http://www.open-mpi.org/__community/lists/pmix-devel/__2014/11/0293.php
> >>> <
> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>>
> >>> <
> http://www.open-mpi.org/__community/lists/pmix-devel/__2014/11/0293.php
> >>> <
> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>>
> >>> <
> http://www.open-mpi.org/__community/lists/pmix-devel/__2014/11/0293.php
> >>> <
> http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php>>
> >>>
> >>>
> >>> bottom line, i agree this is a
> >>> slurm issue (slurm plugin should
> >>> depend
> >>> on libslurm, but they do not,
> yet)
> >>>
> >>> a possible workaround would be
> >>> to make the pmi component a
> >>> "proxy" that
> >>> dlopen with RTLD_GLOBAL the
> >>> "real" component in which the
> >>> job is done.
> >>> that being said, the impact is
> >>> quite limited (no direct launch
> >>> in slurm
> >>> with pmi1, but pmi2 works fine)
> >>> so it makes sense not to work
> around
> >>> someone else problem.
> >>> and that being said, configure
> >>> could detect this broken pmi1
> >>> and not
> >>> build pmi1 support or print a
> >>> user friendly error message if
> >>> pmi1 is used.
> >>>
> >>> any thoughts ?
> >>>
> >>> Cheers,
> >>>
> >>> Gilles
> >>>
> >>> On 2014/12/02 7:47, Jeff Squyres
> >>> (jsquyres) wrote:
> >>>
> >>> Ok, if the problem is moot,
> >>> great.
> >>>
> >>> (sidenote: this is moot, so
> >>> ignore this if you want:
> >>> with this explanation, I'm
> >>> still not sure how
> >>> RTLD_GLOBAL fixes the issue)
> >>>
> >>>
> >>> On Dec 1, 2014, at 5:15 PM,
> >>> Ralph Castain
> >>> <[email protected]
> >>> <mailto:[email protected]>>
> >>> <mailto:[email protected]
> >>> <mailto:[email protected]>>
> >>> wrote:
> >>>
> >>>
> >>> Easy enough to explain.
> >>> We link libpmi into the
> >>> pmix/s1 component. This
> >>> library is missing the
> >>> linkage to libslurm that
> >>> contains the linkage to
> >>> libauth where munge
> >>> resides. So when we call
> >>> a PMI function, libpmi
> >>> references a call to
> >>> munge for authentication
> >>> and hits an “unresolved
> >>> symbol” error.
> >>>
> >>> Moe acknowledges the
> >>> error is in Slurm and is
> >>> fixing the linkages so
> >>> this problem goes away
> >>>
> >>>
> >>>
> >>> On Dec 1, 2014, at
> >>> 2:13 PM, Jeff
> >>> Squyres (jsquyres)
> >>> <[email protected]
> >>> <mailto:
> [email protected]>>
> >>> <mailto:
> [email protected]
> >>> <mailto:
> [email protected]>>
> >>> wrote:
> >>>
> >>> On Dec 1, 2014, at
> >>> 5:07 PM, Ralph
> Castain
> >>> <[email protected]
> >>> <mailto:
> [email protected]>>
> >>> <mailto:
> [email protected]
> >>> <mailto:
> [email protected]>>
> >>> wrote:
> >>>
> >>>
> >>> FWIW: It’s
> >>> Slurm’s pmi-1
> >>> library that
> >>> isn’t linked
> >>> correctly
> >>> against its
> >>> dependencies
> >>> (the pmi-2 one
> >>> is correct).
> >>> Moe is aware of
> >>> the problem and
> >>> fixing it on
> >>> their side. This
> >>> won’t help
> >>> existing
> >>> installations
> >>> until they
> >>> upgrade, but I
> >>> tend to agree
> >>> with Jeff about
> >>> not fixing other
> >>> people’s
> problems.
> >>>
> >>> Can you explain what
> >>> is happening?
> >>>
> >>> I ask because I'm
> >>> not sure I
> >>> understand the
> >>> problem such that
> >>> using RTLD_GLOBAL
> >>> would fix it. I.e.,
> >>> even if libpmi1.so
> >>> isn't linked against
> >>> its dependencies
> >>> properly, that
> >>> shouldn't cause a
> >>> problem if OMPI
> >>> components A and B
> >>> are both linked
> >>> against libpmi1.so,
> >>> and then A is
> >>> loaded, and then B
> >>> is loaded.
> >>>
> >>> ...or perhaps we can
> >>> just discuss this on
> >>> the call tomorrow?
> >>>
> >>> --
> >>> Jeff Squyres
> >>>
> >>> [email protected]
> >>> <mailto:
> [email protected]>
> >>> <mailto:
> [email protected]
> >>> <mailto:
> [email protected]>>
> >>>
> >>> For corporate legal
> >>> information go to:
> >>>
> http://www.cisco.com/web/__about/doing_business/legal/__cri/
> >>> <
> http://www.cisco.com/web/about/doing_business/legal/cri/>
> >>> <
> http://www.cisco.com/web/__about/doing_business/legal/__cri/
> >>> <
> http://www.cisco.com/web/about/doing_business/legal/cri/>>
> >>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected]
> >>> <mailto:
> [email protected]>
> >>> <mailto:
> [email protected]
> >>> <mailto:
> [email protected]>>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16383.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16383.php>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16383.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16383.php>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected]
> >>> <mailto:
> [email protected]>
> >>> <mailto:
> [email protected]
> >>> <mailto:
> [email protected]>>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16384.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16384.php>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16384.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16384.php>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected]
> >>> <mailto:[email protected]>
> >>> <mailto:[email protected]
> >>> <mailto:[email protected]>>
> >>> <mailto:[email protected]
> >>> <mailto:[email protected]>>
> >>> <mailto:[email protected]
> >>> <mailto:[email protected]>>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16386.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16386.php>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16386.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16386.php>>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16386.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16386.php>>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16386.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16386.php>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected]
> >>> <mailto:[email protected]>
> >>> <mailto:[email protected]
> >>> <mailto:[email protected]>>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> <
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16387.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16387.php>
> >>> <
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16387.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16387.php>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected]
> >>> <mailto:[email protected]>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16388.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16388.php>
> >>>
> >>>
> >>>
> >>>
> _________________________________________________
> >>> devel mailing list
> >>>
> >>> [email protected] <mailto:
> [email protected]>
> >>>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16389.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16389.php>
> >>>
> >>>
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>>
> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <
> http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16390.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16390.php>
> >>>
> >>>
> >>>
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16391.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16391.php>
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16393.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16393.php>
> >>>
> >>>
> >>>
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16395.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16395.php>
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16396.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16396.php>
> >>>
> >>>
> >>>
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16397.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16397.php>
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription:
> >>> http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>>
> http://www.open-mpi.org/__community/lists/devel/2014/12/__16398.php
> >>> <
> http://www.open-mpi.org/community/lists/devel/2014/12/16398.php>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Edgar Gabriel
> >>> Associate Professor
> >>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
> >>> Department of Computer Science University of Houston
> >>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
> >>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
> >>> _________________________________________________
> >>> devel mailing list
> >>> [email protected] <mailto:[email protected]>
> >>> Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/devel
> >>> <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> >>> Link to this post:
> >>> http://www.open-mpi.org/__community/lists/devel/2014/12/__16400.php
> >>> <http://www.open-mpi.org/community/lists/devel/2014/12/16400.php>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>>
> >>>
> >>> _______________________________________________
> >>> devel mailing list
> >>> [email protected]
> >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16404.php
> >>>
> >>
> >> --
> >> Edgar Gabriel
> >> Associate Professor
> >> Parallel Software Technologies Lab http://pstl.cs.uh.edu
> >> Department of Computer Science University of Houston
> >> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
> >> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
> >> _______________________________________________
> >> devel mailing list
> >> [email protected]
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16405.php
> >
> >
> > --
> > Jeff Squyres
> > [email protected]
> > For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >
>
>
> --
> Jeff Squyres
> [email protected]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> _______________________________________________
> devel mailing list
> [email protected]
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16409.php
--
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
