[OMPI devel] SLURM affinity accounting in Open MPI

[Artem Polyakov]

Hello

I found that SLURM installations that use cgroup plugin and
have TaskAffinity=yes in cgroup.conf have problems with OpenMPI: all
processes on non-launch node are assigned on one core. This leads to quite
poor performance.
The problem can be seen only if using mpirun to start parallel application
in batch script. For example: *mpirun ./mympi*
If using srun with PMI affinity is setted properly: *srun ./mympi.*

Close look shows that the reason lies in the way Open MPI use srun to
launch ORTE daemons. Here is example of the command line:
*srun* *--nodes=1* *--ntasks=1* --kill-on-bad-exit --nodelist=node02
*orted*-mca ess slurm -mca orte_ess_jobid 3799121920 -mca
orte_ess_vpid

Saying *--nodes=1* *--ntasks=1* to SLURM means that you want to start one
task and (with TaskAffinity=yes) it will be binded to one core. Next orted
use this affinity as base for all spawned branch processes. If I understand
correctly the problem behind using srun is that if you say *srun*
*--nodes=1* *--ntasks=4* - then SLURM will spawn 4 independent orted
processes binded to different cores which is not what we really need.

I found that disabling of cpu binding as a fast hack works good for cgroup
plugin. Since job runs inside a group which has core access restrictions,
spawned branch processes are executed under nodes scheduler control on all
allocated cores. The command line looks like this:
srun *--cpu_bind=none* --nodes=1 --ntasks=1 --kill-on-bad-exit
--nodelist=node02 orted -mca ess slurm -mca orte_ess_jobid 3799121920 -mca
orte_ess_vpid

This solution will probably won't work with SLURM task/affinity plugin.
Also it may be a bad idea when strong affinity desirable.

My patch to stable Open MPI version (1.6.5) is attached to this e-mail. I
will try to make more reliable solution but I need more time and beforehand
would like to know opinion of Open MPI developers.

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
diff -Naur openmpi-1.6.5-old/orte/mca/plm/slurm/plm_slurm_module.c openmpi-1.6.5-new/orte/mca/plm/slurm/plm_slurm_module.c
--- openmpi-1.6.5-old/orte/mca/plm/slurm/plm_slurm_module.c	2012-04-03 10:30:29.0 -0400
+++ openmpi-1.6.5-new/orte/mca/plm/slurm/plm_slurm_module.c	2014-02-12 03:59:23.763664648 -0500
@@ -257,6 +257,8 @@
 
 /* add the srun command */
 opal_argv_append(&argc, &argv, "srun");
+opal_argv_append(&argc, &argv, "--cpu_bind=none");
 
 /* Append user defined arguments to srun */
 if ( NULL != mca_plm_slurm_component.custom_args ) {

Re: [OMPI devel] SLURM affinity accounting in Open MPI

[Artem Polyakov]

Hello,

I think I am ready to return to mpirun affinity handling discussion. I have
more general solution now. It is beta-tested (just one cluster with SLURM
using cgroups plugin). But it shows my main idea and if it is worth to be
included into mainstream I am ready to polish or improove it.

The code respects SLURM cpu allocation through SLURM_JOB_CPUS_PER_NODE and
handles such cases correctly:
SLURM_JOB_CPUS_PER_NODE='12(x3),7'
SLURM_JOB_NODELIST=node[0-3]

It splits the node lists into groups having equal number of cpus. In the
example above we will have 2 groups:
1) node0, node1, node2 with 12 cpus
2) node3 with 7 cpus.

then it uses separate srun's for each group.

The weakness of this patch is that we need to deal with several srun's and
I am not sure that cleanup will perform correctly. I plan to test this case
additionaly.


2014-02-12 17:42 GMT+07:00 Artem Polyakov :

> Hello
>
> I found that SLURM installations that use cgroup plugin and
> have TaskAffinity=yes in cgroup.conf have problems with OpenMPI: all
> processes on non-launch node are assigned on one core. This leads to quite
> poor performance.
> The problem can be seen only if using mpirun to start parallel application
> in batch script. For example: *mpirun ./mympi*
> If using srun with PMI affinity is setted properly: *srun ./mympi.*
>
> Close look shows that the reason lies in the way Open MPI use srun to
> launch ORTE daemons. Here is example of the command line:
> *srun* *--nodes=1* *--ntasks=1* --kill-on-bad-exit --nodelist=node02
> *orted* -mca ess slurm -mca orte_ess_jobid 3799121920 -mca orte_ess_vpid
>
> Saying *--nodes=1* *--ntasks=1* to SLURM means that you want to start one
> task and (with TaskAffinity=yes) it will be binded to one core. Next orted
> use this affinity as base for all spawned branch processes. If I understand
> correctly the problem behind using srun is that if you say *srun*
> *--nodes=1* *--ntasks=4* - then SLURM will spawn 4 independent orted
> processes binded to different cores which is not what we really need.
>
> I found that disabling of cpu binding as a fast hack works good for cgroup
> plugin. Since job runs inside a group which has core access restrictions,
> spawned branch processes are executed under nodes scheduler control on all
> allocated cores. The command line looks like this:
> srun *--cpu_bind=none* --nodes=1 --ntasks=1 --kill-on-bad-exit
> --nodelist=node02 orted -mca ess slurm -mca orte_ess_jobid 3799121920 -mca
> orte_ess_vpid
>
> This solution will probably won't work with SLURM task/affinity plugin.
> Also it may be a bad idea when strong affinity desirable.
>
> My patch to stable Open MPI version (1.6.5) is attached to this e-mail. I
> will try to make more reliable solution but I need more time and beforehand
> would like to know opinion of Open MPI developers.
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
diff -Naur openmpi-1.6.5/orte/mca/plm/slurm/plm_slurm_module.c openmpi-1.6.5_new/orte/mca/plm/slurm/plm_slurm_module.c
--- openmpi-1.6.5/orte/mca/plm/slurm/plm_slurm_module.c	2012-04-03 10:30:29.0 -0400
+++ openmpi-1.6.5_new/orte/mca/plm/slurm/plm_slurm_module.c	2014-03-07 08:54:12.878010513 -0500
@@ -11,7 +11,7 @@
  * All rights reserved.
  * Copyright (c) 2006-2007 Cisco Systems, Inc.  All rights reserved.
  * Copyright (c) 2007  Los Alamos National Security, LLC.  All rights
- * reserved. 
+ * reserved.
  * $COPYRIGHT$
  *
  * Additional copyrights may follow
@@ -107,12 +107,26 @@
 /*
  * Local variables
  */
-static pid_t primary_srun_pid = 0;
-static bool primary_pid_set = false;
+static pid_t *primary_srun_pids = NULL;
+static int primary_srun_pids_cnt = 0;
+static bool primary_pids_set = false;
 static orte_jobid_t active_job = ORTE_JOBID_INVALID;
 static bool launching_daemons;
 static bool local_launch_available = false;
 
+struct pml_slurm_node_group_t {
+char **nodelist;
+unsigned int nodelist_size;
+unsigned int cpus_per_node;
+};
+
+static int plm_slurm_group_nodes(orte_job_map_t *map,
+ struct pml_slurm_node_group_t **grpout,
+ int *grpcnt);
+static void plm_slurm_groups_free(struct pml_slurm_node_group_t **groups,
+  int grpcnt);
+
+
 /**
 * Init the module
  */
@@ -152,7 +166,7 @@
 char *tmp;
 char** env = NULL;
 char* var;
-char *nodelist_flat;
+char *nodelist_flat, *nodelist_full;
 char **nodelist_argv;
 char *name_string;
 char **custom_strings;
@@ -163,11 +177,14 @@
 orte_jobid_t failed_job;
 bool failed_launch=true;
 bool using_regexp=false;
+struct pml_sl

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

Good idea :)!

среда, 7 мая 2014 г. пользователь Ralph Castain написал:

> Jeff actually had a useful suggestion (gasp!).He proposed that we separate
> the PMI-1 and PMI-2 codes into separate components so you could select them
> at runtime. Thus, we would build both (assuming both PMI-1 and 2 libs are
> found), default to PMI-1, but users could select to try PMI-2. If the PMI-2
> component failed, we would emit a show_help indicating that they probably
> have a broken PMI-2 version and should try PMI-1.
>
> Make sense?
> Ralph
>
> On May 7, 2014, at 8:00 AM, Ralph Castain  wrote:
>
>
> On May 7, 2014, at 7:56 AM, Joshua Ladd  wrote:
>
> Ah, I see. Sorry for the reactionary comment - but this feature falls
> squarely within my "jurisdiction", and we've invested a lot in improving
> OMPI jobstart under srun.
>
> That being said (now that I've taken some deep breaths and carefully read
> your original email :)), what you're proposing isn't a bad idea. I think it
> would be good to maybe add a "--with-pmi2" flag to configure since
> "--with-pmi" automagically uses PMI2 if it finds the header and lib. This
> way, we could experiment with PMI1/PMI2 without having to rebuild SLURM or
> hack the installation.
>
>
> That would be a much simpler solution than what Artem proposed (off-list)
> where we would try PMI2 and then if it didn't work try to figure out how to
> fall back to PMI1. I'll add this for now, and if Artem wants to try his
> more automagic solution and can make it work, then we can reconsider that
> option.
>
> Thanks
> Ralph
>
>
> Josh
>
>
> On Wed, May 7, 2014 at 10:45 AM, Ralph Castain  wrote:
>
> Okay, then we'll just have to develop a workaround for all those Slurm
> releases where PMI-2 is borked :-(
>
> FWIW: I think people misunderstood my statement. I specifically did *not*
> propose to *lose* PMI-2 support. I suggested that we change it to
> "on-by-request" instead of the current "on-by-default" so we wouldn't keep
> getting asked about PMI-2 bugs in Slurm. Once the Slurm implementation
> stabilized, then we could reverse that policy.
>
> However, given that both you and Chris appear to prefer to keep it
> "on-by-default", we'll see if we can find a way to detect that PMI-2 is
> broken and then fall back to PMI-1.
>
>
> On May 7, 2014, at 7:39 AM, Joshua Ladd  wrote:
>
> Just saw this thread, and I second Chris' observations: at scale we are
> seeing huge gains in jobstart performance with PMI2 over PMI1. We 
> *CANNOT*loose this functionality. For competitive reasons, I cannot provide 
> exact
> numbers, but let's say the difference is in the ballpark of a full
> order-of-magnitude on 20K ranks versus PMI1. PMI1 is completely
> unacceptable/unusable at scale. Certainly PMI2 still has scaling issues,
> but there is no contest between PMI1 and PMI2.  We (MLNX) are actively
> working to resolve some of the scalability issues in PMI2.
>
> Josh
>
> Joshua S. Ladd
> Staff Engineer, HPC Software
> Mellanox Technologies
>
> Email: josh...@mellanox.com
>
>
> On Wed, May 7, 2014 at 4:00 AM, Ralph Castain  wrote:
>
> Interesting - how many nodes were involved? As I said, the bad scaling
> becomes more evident at a fairly high node count.
>
> On May 7, 2014, at 12:07 AM, Christopher Samuel 
> wrote:
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Hiya Ralph,
> >
> > On 07/05/14 14:49, Ralph Castain wrote:
> >
> >> I should have looked closer to see the numbers you posted, Chris -
> >> those include time for MPI wireup. So what you are seeing is that
> >> mpirun is much more efficient at exchanging the MPI endpoint info
> >> than PMI. I suspect that PMI2 is not much better as the primary
> >> reason for the difference is that mpriun sends blobs, while PMI
> >> requires that everything b
>
>

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

That's a good point. There is actually a bunch of modules in ompi, opal and
orte that has to be duplicated.

среда, 7 мая 2014 г. пользователь Joshua Ladd написал:

> +1 Sounds like a good idea - but decoupling the two and adding all the
> right selection mojo might be a bit of a pain. There are several places in
> OMPI where the distinction between PMI1and PMI2 is made, not only in
> grpcomm. DB and ESS frameworks off the top of my head.
>
> Josh
>
>
> On Wed, May 7, 2014 at 11:48 AM, Artem Polyakov wrote:
>
>> Good idea :)!
>>
>> среда, 7 мая 2014 г. пользователь Ralph Castain написал:
>>
>> Jeff actually had a useful suggestion (gasp!).He proposed that we
>> separate the PMI-1 and PMI-2 codes into separate components so you could
>> select them at runtime. Thus, we would build both (assuming both PMI-1 and
>> 2 libs are found), default to PMI-1, but users could select to try PMI-2.
>> If the PMI-2 component failed, we would emit a show_help indicating that
>> they probably have a broken PMI-2 version and should try PMI-1.
>>
>> Make sense?
>> Ralph
>>
>> On May 7, 2014, at 8:00 AM, Ralph Castain  wrote:
>>
>>
>> On May 7, 2014, at 7:56 AM, Joshua Ladd  wrote:
>>
>> Ah, I see. Sorry for the reactionary comment - but this feature falls
>> squarely within my "jurisdiction", and we've invested a lot in improving
>> OMPI jobstart under srun.
>>
>> That being said (now that I've taken some deep breaths and carefully read
>> your original email :)), what you're proposing isn't a bad idea. I think it
>> would be good to maybe add a "--with-pmi2" flag to configure since
>> "--with-pmi" automagically uses PMI2 if it finds the header and lib. This
>> way, we could experiment with PMI1/PMI2 without having to rebuild SLURM or
>> hack the installation.
>>
>>
>> That would be a much simpler solution than what Artem proposed (off-list)
>> where we would try PMI2 and then if it didn't work try to figure out how to
>> fall back to PMI1. I'll add this for now, and if Artem wants to try his
>> more automagic solution and can make it work, then we can reconsider that
>> option.
>>
>> Thanks
>> Ralph
>>
>>
>> Josh
>>
>>
>> On Wed, May 7, 2014 at 10:45 AM, Ralph Castain  wrote:
>>
>> Okay, then we'll just have to develop a workaround for all those Slurm
>> releases where PMI-2 is borked :-(
>>
>> FWIW: I think people misunderstood my statement. I specifically did *not*
>> propose to *lose* PMI-2 support. I suggested that we change it to
>> "on-by-request" instead of the current "on-by-default" so we wouldn't keep
>> getting asked about PMI-2 bugs in Slurm. Once the Slurm implementation
>> stabilized, then we could reverse that policy.
>>
>> However, given that both you and Chris appear to prefer to keep it
>> "on-by-default", we'll see if we can find a way to detect that PMI-2 is
>> broken and then fall back to PMI-1.
>>
>>
>> On May 7, 2014, at 7:39 AM, Joshua Ladd  wrote:
>>
>> Just saw this thread, and I second Chris' observations: at scale we are
>> seeing huge gains in jobstart performance with PMI2 over PMI1. We
>> *CANNOT* loose this functionality. For competitive reasons, I cannot
>> provide exact numbers, but let's say the difference is in the ballpark of a
>> full order-of-magnitude on 20K ranks versus PMI1. PMI1 is completely
>> unacceptable/unusable at scale. Certainly PMI2 still has scaling issues,
>> but there is no contest between PMI1 and PMI2.  We (MLNX) are actively
>> working to resolve some of the scalability issues in PMI2.
>>
>> Josh
>>
>> Joshua S. Ladd
>> Staff Engineer, HPC Software
>> Mellanox Technologies
>>
>> Email: josh...@mellanox.com
>>
>>
>> On Wed, May 7, 2014 at 4:00 AM, Ralph Castain  wrote:
>>
>> Interesting - how many nodes were involved? As I said, the bad scaling
>> becomes more evident at a fairly high node count.
>>
>> On May 7, 2014, at 12:07 AM, Christopher Samuel 
>> wrote:
>>
>> > -BEGIN PGP SIGNED MESSAGE-
>> > Hash: SHA1
>> >
>> > Hiya Ralph,
>> >
>> > On 07/05/14 14:49, Ralph Castain wrote:
>> >
>> >> I should have looked closer to see the numbers you posted, Chris -
>> >> those include time for MPI wireup. So what you are seeing is that
>> >> mpirun is much more efficient at exchanging the MPI endpoint info
>> >> than PMI. I suspect that PMI2 is not much better as the primary
>> >> reason for the difference is that mpriun sends blobs, while PMI
>> >> requires that everything b
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/05/14716.php
>>
>
>

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

We discussed with Ralph Joshuas concerns and decided to try automatic PMI2
correctness first as it was initially intended. Here is my idea. The
universal way to decide if PMI2 is correct is to compare PMI_Init(..,
&rank, &size, ...) and PMI2_Init(.., &rank, &size, ...). Size and rank
should be equal. In this case we proceed with PMI2 finalizing PMI1.
Otherwise we finalize PMI2 and proceed with PMI1.
I need to clarify with SLURM guys if parallel initialization of both PMIs
are legal. If not - we'll do that sequentially.
In other places we'll just use the flag saying what PMI version to use.
Does that sounds reasonable?

2014-05-07 23:10 GMT+07:00 Artem Polyakov :

> That's a good point. There is actually a bunch of modules in ompi, opal
> and orte that has to be duplicated.
>
> среда, 7 мая 2014 г. пользователь Joshua Ladd написал:
>
>>  +1 Sounds like a good idea - but decoupling the two and adding all the
>> right selection mojo might be a bit of a pain. There are several places in
>> OMPI where the distinction between PMI1and PMI2 is made, not only in
>> grpcomm. DB and ESS frameworks off the top of my head.
>>
>> Josh
>>
>>
>> On Wed, May 7, 2014 at 11:48 AM, Artem Polyakov wrote:
>>
>>> Good idea :)!
>>>
>>> среда, 7 мая 2014 г. пользователь Ralph Castain написал:
>>>
>>> Jeff actually had a useful suggestion (gasp!).He proposed that we
>>> separate the PMI-1 and PMI-2 codes into separate components so you could
>>> select them at runtime. Thus, we would build both (assuming both PMI-1 and
>>> 2 libs are found), default to PMI-1, but users could select to try PMI-2.
>>> If the PMI-2 component failed, we would emit a show_help indicating that
>>> they probably have a broken PMI-2 version and should try PMI-1.
>>>
>>> Make sense?
>>> Ralph
>>>
>>> On May 7, 2014, at 8:00 AM, Ralph Castain  wrote:
>>>
>>>
>>> On May 7, 2014, at 7:56 AM, Joshua Ladd  wrote:
>>>
>>> Ah, I see. Sorry for the reactionary comment - but this feature falls
>>> squarely within my "jurisdiction", and we've invested a lot in improving
>>> OMPI jobstart under srun.
>>>
>>> That being said (now that I've taken some deep breaths and carefully
>>> read your original email :)), what you're proposing isn't a bad idea. I
>>> think it would be good to maybe add a "--with-pmi2" flag to configure since
>>> "--with-pmi" automagically uses PMI2 if it finds the header and lib. This
>>> way, we could experiment with PMI1/PMI2 without having to rebuild SLURM or
>>> hack the installation.
>>>
>>>
>>> That would be a much simpler solution than what Artem proposed
>>> (off-list) where we would try PMI2 and then if it didn't work try to figure
>>> out how to fall back to PMI1. I'll add this for now, and if Artem wants to
>>> try his more automagic solution and can make it work, then we can
>>> reconsider that option.
>>>
>>> Thanks
>>> Ralph
>>>
>>>
>>> Josh
>>>
>>>
>>> On Wed, May 7, 2014 at 10:45 AM, Ralph Castain  wrote:
>>>
>>> Okay, then we'll just have to develop a workaround for all those Slurm
>>> releases where PMI-2 is borked :-(
>>>
>>> FWIW: I think people misunderstood my statement. I specifically did
>>> *not* propose to *lose* PMI-2 support. I suggested that we change it to
>>> "on-by-request" instead of the current "on-by-default" so we wouldn't keep
>>> getting asked about PMI-2 bugs in Slurm. Once the Slurm implementation
>>> stabilized, then we could reverse that policy.
>>>
>>> However, given that both you and Chris appear to prefer to keep it
>>> "on-by-default", we'll see if we can find a way to detect that PMI-2 is
>>> broken and then fall back to PMI-1.
>>>
>>>
>>> On May 7, 2014, at 7:39 AM, Joshua Ladd  wrote:
>>>
>>> Just saw this thread, and I second Chris' observations: at scale we are
>>> seeing huge gains in jobstart performance with PMI2 over PMI1. We
>>> *CANNOT* loose this functionality. For competitive reasons, I cannot
>>> provide exact numbers, but let's say the difference is in the ballpark of a
>>> full order-of-magnitude on 20K ranks versus PMI1. PMI1 is completely
>>> unacceptable/unusable at scale. Certainly PMI2 still has scaling issues,
>>> but there is no contest between PMI1 an

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

2014-05-08 5:54 GMT+07:00 Ralph Castain :

> Ummmno, I don't think that's right. I believe we decided to instead
> create the separate components, default to PMI-2 if available, print nice
> error message if not, otherwise use PMI-1.
>
> I don't want to initialize both PMIs in parallel as most installations
> won't support it.
>

Ok, I agree. Beside the lack of support there can be a performance hit
caused by PMI1 initialization at scale. This is not a case of SLURM PMI1
since it is quite simple and local. But I didn't consider other
implementations.

On May 7, 2014, at 3:49 PM, Artem Polyakov  wrote:
>
> We discussed with Ralph Joshuas concerns and decided to try automatic PMI2
> correctness first as it was initially intended. Here is my idea. The
> universal way to decide if PMI2 is correct is to compare PMI_Init(..,
> &rank, &size, ...) and PMI2_Init(.., &rank, &size, ...). Size and rank
> should be equal. In this case we proceed with PMI2 finalizing PMI1.
> Otherwise we finalize PMI2 and proceed with PMI1.
> I need to clarify with SLURM guys if parallel initialization of both PMIs
> are legal. If not - we'll do that sequentially.
> In other places we'll just use the flag saying what PMI version to use.
> Does that sounds reasonable?
>
> 2014-05-07 23:10 GMT+07:00 Artem Polyakov :
>
>> That's a good point. There is actually a bunch of modules in ompi, opal
>> and orte that has to be duplicated.
>>
>> среда, 7 мая 2014 г. пользователь Joshua Ladd написал:
>>
>>> +1 Sounds like a good idea - but decoupling the two and adding all the
>>> right selection mojo might be a bit of a pain. There are several places in
>>> OMPI where the distinction between PMI1and PMI2 is made, not only in
>>> grpcomm. DB and ESS frameworks off the top of my head.
>>>
>>> Josh
>>>
>>>
>>> On Wed, May 7, 2014 at 11:48 AM, Artem Polyakov 
>>> wrote:
>>>
>>>> Good idea :)!
>>>>
>>>> среда, 7 мая 2014 г. пользователь Ralph Castain написал:
>>>>
>>>> Jeff actually had a useful suggestion (gasp!).He proposed that we
>>>> separate the PMI-1 and PMI-2 codes into separate components so you could
>>>> select them at runtime. Thus, we would build both (assuming both PMI-1 and
>>>> 2 libs are found), default to PMI-1, but users could select to try PMI-2.
>>>> If the PMI-2 component failed, we would emit a show_help indicating that
>>>> they probably have a broken PMI-2 version and should try PMI-1.
>>>>
>>>> Make sense?
>>>> Ralph
>>>>
>>>> On May 7, 2014, at 8:00 AM, Ralph Castain  wrote:
>>>>
>>>>
>>>> On May 7, 2014, at 7:56 AM, Joshua Ladd  wrote:
>>>>
>>>> Ah, I see. Sorry for the reactionary comment - but this feature falls
>>>> squarely within my "jurisdiction", and we've invested a lot in improving
>>>> OMPI jobstart under srun.
>>>>
>>>> That being said (now that I've taken some deep breaths and carefully
>>>> read your original email :)), what you're proposing isn't a bad idea. I
>>>> think it would be good to maybe add a "--with-pmi2" flag to configure since
>>>> "--with-pmi" automagically uses PMI2 if it finds the header and lib. This
>>>> way, we could experiment with PMI1/PMI2 without having to rebuild SLURM or
>>>> hack the installation.
>>>>
>>>>
>>>> That would be a much simpler solution than what Artem proposed
>>>> (off-list) where we would try PMI2 and then if it didn't work try to figure
>>>> out how to fall back to PMI1. I'll add this for now, and if Artem wants to
>>>> try his more automagic solution and can make it work, then we can
>>>> reconsider that option.
>>>>
>>>> Thanks
>>>> Ralph
>>>>
>>>>
>>>> Josh
>>>>
>>>>
>>>> On Wed, May 7, 2014 at 10:45 AM, Ralph Castain 
>>>> wrote:
>>>>
>>>> Okay, then we'll just have to develop a workaround for all those Slurm
>>>> releases where PMI-2 is borked :-(
>>>>
>>>> FWIW: I think people misunderstood my statement. I specifically did
>>>> *not* propose to *lose* PMI-2 support. I suggested that we change it to
>>>> "on-by-request" instead of the current "on-by-default" so we wouldn't keep
>>>> getting a

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

Just reread your suggestions in our out-of-list discussion and found that I
misunderstand it. So no parallel PMI! Take all possible code into
opal/mca/common/pmi.

To additionally clarify what is the preferred way:
1. to create one joined PMI module having a switches to decide what
functiononality to implement.
2. or to have 2 separate common modules for PMI1 and one for PMI2, and does
this fit opal/mca/common/ ideology at all?


2014-05-08 6:44 GMT+07:00 Artem Polyakov :

>
> 2014-05-08 5:54 GMT+07:00 Ralph Castain :
>
> Ummmno, I don't think that's right. I believe we decided to instead
>> create the separate components, default to PMI-2 if available, print nice
>> error message if not, otherwise use PMI-1.
>>
>> I don't want to initialize both PMIs in parallel as most installations
>> won't support it.
>>
>
> Ok, I agree. Beside the lack of support there can be a performance hit
> caused by PMI1 initialization at scale. This is not a case of SLURM PMI1
> since it is quite simple and local. But I didn't consider other
> implementations.
>
> On May 7, 2014, at 3:49 PM, Artem Polyakov  wrote:
>>
>> We discussed with Ralph Joshuas concerns and decided to try automatic
>> PMI2 correctness first as it was initially intended. Here is my idea. The
>> universal way to decide if PMI2 is correct is to compare PMI_Init(..,
>> &rank, &size, ...) and PMI2_Init(.., &rank, &size, ...). Size and rank
>> should be equal. In this case we proceed with PMI2 finalizing PMI1.
>> Otherwise we finalize PMI2 and proceed with PMI1.
>> I need to clarify with SLURM guys if parallel initialization of both PMIs
>> are legal. If not - we'll do that sequentially.
>> In other places we'll just use the flag saying what PMI version to use.
>> Does that sounds reasonable?
>>
>> 2014-05-07 23:10 GMT+07:00 Artem Polyakov :
>>
>>> That's a good point. There is actually a bunch of modules in ompi, opal
>>> and orte that has to be duplicated.
>>>
>>> среда, 7 мая 2014 г. пользователь Joshua Ladd написал:
>>>
>>>> +1 Sounds like a good idea - but decoupling the two and adding all the
>>>> right selection mojo might be a bit of a pain. There are several places in
>>>> OMPI where the distinction between PMI1and PMI2 is made, not only in
>>>> grpcomm. DB and ESS frameworks off the top of my head.
>>>>
>>>> Josh
>>>>
>>>>
>>>> On Wed, May 7, 2014 at 11:48 AM, Artem Polyakov 
>>>> wrote:
>>>>
>>>>> Good idea :)!
>>>>>
>>>>> среда, 7 мая 2014 г. пользователь Ralph Castain написал:
>>>>>
>>>>> Jeff actually had a useful suggestion (gasp!).He proposed that we
>>>>> separate the PMI-1 and PMI-2 codes into separate components so you could
>>>>> select them at runtime. Thus, we would build both (assuming both PMI-1 and
>>>>> 2 libs are found), default to PMI-1, but users could select to try PMI-2.
>>>>> If the PMI-2 component failed, we would emit a show_help indicating that
>>>>> they probably have a broken PMI-2 version and should try PMI-1.
>>>>>
>>>>> Make sense?
>>>>> Ralph
>>>>>
>>>>> On May 7, 2014, at 8:00 AM, Ralph Castain  wrote:
>>>>>
>>>>>
>>>>> On May 7, 2014, at 7:56 AM, Joshua Ladd  wrote:
>>>>>
>>>>> Ah, I see. Sorry for the reactionary comment - but this feature falls
>>>>> squarely within my "jurisdiction", and we've invested a lot in improving
>>>>> OMPI jobstart under srun.
>>>>>
>>>>> That being said (now that I've taken some deep breaths and carefully
>>>>> read your original email :)), what you're proposing isn't a bad idea. I
>>>>> think it would be good to maybe add a "--with-pmi2" flag to configure 
>>>>> since
>>>>> "--with-pmi" automagically uses PMI2 if it finds the header and lib. This
>>>>> way, we could experiment with PMI1/PMI2 without having to rebuild SLURM or
>>>>> hack the installation.
>>>>>
>>>>>
>>>>> That would be a much simpler solution than what Artem proposed
>>>>> (off-list) where we would try PMI2 and then if it didn't work try to 
>>>>> figure
>>>>> out how to fall back to PMI1. I'll add this for now, and if Artem wants to
>&

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

I like #2 too.
But my question was slightly different. Can we incapsulate PMI logic that
OMPI use in common/pmi as #2 suggests but have 2 different implementations
of this component say common/pmi and common/pmi2? I am asking because I
have concerns that this kind of component is not supposed to be duplicated.
In this case we could have one common MCA parameter and 2 components as it
was suggested by Jeff.


2014-05-08 7:01 GMT+07:00 Ralph Castain :

> The desired solution is to have the ability to select pmi-1 vs pmi-2 at
> runtime. This can be done in two ways:
>
> 1. you could have separate pmi1 and pmi2 components in each framework.
> You'd want to define only one common MCA param to direct the selection,
> however.
>
> 2. you could have a single pmi component in each framework, calling code
> in the appropriate common/pmi location. You would then need a runtime MCA
> param to select whether pmi-1 or pmi-2 was going to be used, and have the
> common code check before making the desired calls.
>
> The choice of method is left up to you. They each have their negatives. If
> it were me, I'd probably try #2 first, assuming the codes are mostly common
> in the individual frameworks.
>
>
> On May 7, 2014, at 4:51 PM, Artem Polyakov  wrote:
>
> Just reread your suggestions in our out-of-list discussion and found that
> I misunderstand it. So no parallel PMI! Take all possible code into
> opal/mca/common/pmi.
> To additionally clarify what is the preferred way:
> 1. to create one joined PMI module having a switches to decide what
> functiononality to implement.
> 2. or to have 2 separate common modules for PMI1 and one for PMI2, and
> does this fit opal/mca/common/ ideology at all?
>
>
> 2014-05-08 6:44 GMT+07:00 Artem Polyakov :
>
>>
>> 2014-05-08 5:54 GMT+07:00 Ralph Castain :
>>
>> Ummmno, I don't think that's right. I believe we decided to instead
>>> create the separate components, default to PMI-2 if available, print nice
>>> error message if not, otherwise use PMI-1.
>>>
>>> I don't want to initialize both PMIs in parallel as most installations
>>> won't support it.
>>>
>>
>> Ok, I agree. Beside the lack of support there can be a performance hit
>> caused by PMI1 initialization at scale. This is not a case of SLURM PMI1
>> since it is quite simple and local. But I didn't consider other
>> implementations.
>>
>> On May 7, 2014, at 3:49 PM, Artem Polyakov  wrote:
>>>
>>> We discussed with Ralph Joshuas concerns and decided to try automatic
>>> PMI2 correctness first as it was initially intended. Here is my idea. The
>>> universal way to decide if PMI2 is correct is to compare PMI_Init(..,
>>> &rank, &size, ...) and PMI2_Init(.., &rank, &size, ...). Size and rank
>>> should be equal. In this case we proceed with PMI2 finalizing PMI1.
>>> Otherwise we finalize PMI2 and proceed with PMI1.
>>> I need to clarify with SLURM guys if parallel initialization of both
>>> PMIs are legal. If not - we'll do that sequentially.
>>> In other places we'll just use the flag saying what PMI version to use.
>>> Does that sounds reasonable?
>>>
>>> 2014-05-07 23:10 GMT+07:00 Artem Polyakov :
>>>
>>>> That's a good point. There is actually a bunch of modules in ompi, opal
>>>> and orte that has to be duplicated.
>>>>
>>>> среда, 7 мая 2014 г. пользователь Joshua Ladd написал:
>>>>
>>>>> +1 Sounds like a good idea - but decoupling the two and adding all the
>>>>> right selection mojo might be a bit of a pain. There are several places in
>>>>> OMPI where the distinction between PMI1and PMI2 is made, not only in
>>>>> grpcomm. DB and ESS frameworks off the top of my head.
>>>>>
>>>>> Josh
>>>>>
>>>>>
>>>>> On Wed, May 7, 2014 at 11:48 AM, Artem Polyakov 
>>>>> wrote:
>>>>>
>>>>>> Good idea :)!
>>>>>>
>>>>>> среда, 7 мая 2014 г. пользователь Ralph Castain написал:
>>>>>>
>>>>>> Jeff actually had a useful suggestion (gasp!).He proposed that we
>>>>>> separate the PMI-1 and PMI-2 codes into separate components so you could
>>>>>> select them at runtime. Thus, we would build both (assuming both PMI-1 
>>>>>> and
>>>>>> 2 libs are found), default to PMI-1, but users could select to try PMI-2.
>>>>>> If the PMI-2 component failed, we wo

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

2014-05-08 7:15 GMT+07:00 Ralph Castain :

> Take a look in opal/mca/common/pmi - we already do a bunch of #if PMI2
> stuff in there. All we are talking about doing here is:
>
> * making those selections be runtime based on an MCA param, compiling if
> PMI2 is available but selected at runtime
>
> * moving some additional functions into that code area and out of the
> individual components
>

Ok, that is pretty clear now. And will do exactly #2.
Thank you.


>
>
> On May 7, 2014, at 5:08 PM, Artem Polyakov  wrote:
>
> I like #2 too.
> But my question was slightly different. Can we incapsulate PMI logic that
> OMPI use in common/pmi as #2 suggests but have 2 different
> implementations of this component say common/pmi and common/pmi2? I am
> asking because I have concerns that this kind of component is not supposed
> to be duplicated.
> In this case we could have one common MCA parameter and 2 components as it
> was suggested by Jeff.
>
>
> 2014-05-08 7:01 GMT+07:00 Ralph Castain :
>
>> The desired solution is to have the ability to select pmi-1 vs pmi-2 at
>> runtime. This can be done in two ways:
>>
>> 1. you could have separate pmi1 and pmi2 components in each framework.
>> You'd want to define only one common MCA param to direct the selection,
>> however.
>>
>> 2. you could have a single pmi component in each framework, calling code
>> in the appropriate common/pmi location. You would then need a runtime MCA
>> param to select whether pmi-1 or pmi-2 was going to be used, and have the
>> common code check before making the desired calls.
>>
>> The choice of method is left up to you. They each have their negatives.
>> If it were me, I'd probably try #2 first, assuming the codes are mostly
>> common in the individual frameworks.
>>
>>
>> On May 7, 2014, at 4:51 PM, Artem Polyakov  wrote:
>>
>>  Just reread your suggestions in our out-of-list discussion and found
>> that I misunderstand it. So no parallel PMI! Take all possible code into
>> opal/mca/common/pmi.
>> To additionally clarify what is the preferred way:
>> 1. to create one joined PMI module having a switches to decide what
>> functiononality to implement.
>> 2. or to have 2 separate common modules for PMI1 and one for PMI2, and
>> does this fit opal/mca/common/ ideology at all?
>>
>>
>> 2014-05-08 6:44 GMT+07:00 Artem Polyakov :
>>
>>>
>>> 2014-05-08 5:54 GMT+07:00 Ralph Castain :
>>>
>>> Ummmno, I don't think that's right. I believe we decided to instead
>>>> create the separate components, default to PMI-2 if available, print nice
>>>> error message if not, otherwise use PMI-1.
>>>>
>>>> I don't want to initialize both PMIs in parallel as most installations
>>>> won't support it.
>>>>
>>>
>>> Ok, I agree. Beside the lack of support there can be a performance hit
>>> caused by PMI1 initialization at scale. This is not a case of SLURM PMI1
>>> since it is quite simple and local. But I didn't consider other
>>> implementations.
>>>
>>> On May 7, 2014, at 3:49 PM, Artem Polyakov  wrote:
>>>>
>>>> We discussed with Ralph Joshuas concerns and decided to try automatic
>>>> PMI2 correctness first as it was initially intended. Here is my idea. The
>>>> universal way to decide if PMI2 is correct is to compare PMI_Init(..,
>>>> &rank, &size, ...) and PMI2_Init(.., &rank, &size, ...). Size and rank
>>>> should be equal. In this case we proceed with PMI2 finalizing PMI1.
>>>> Otherwise we finalize PMI2 and proceed with PMI1.
>>>> I need to clarify with SLURM guys if parallel initialization of both
>>>> PMIs are legal. If not - we'll do that sequentially.
>>>> In other places we'll just use the flag saying what PMI version to use.
>>>> Does that sounds reasonable?
>>>>
>>>> 2014-05-07 23:10 GMT+07:00 Artem Polyakov :
>>>>
>>>>> That's a good point. There is actually a bunch of modules in ompi,
>>>>> opal and orte that has to be duplicated.
>>>>>
>>>>> среда, 7 мая 2014 г. пользователь Joshua Ladd написал:
>>>>>
>>>>>> +1 Sounds like a good idea - but decoupling the two and adding all
>>>>>> the right selection mojo might be a bit of a pain. There are several 
>>>>>> places
>>>>>> in OMPI where the distinction between PMI1and PMI2 is made, not on

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

Hi Chris.

Current disign is to provide the runtime parameter for PMI version
selection. It would be even more flexible that configuration-time selection
and (with my current understanding) not very hard to acheive.


2014-05-08 8:15 GMT+07:00 Christopher Samuel :

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi all,
>
> Apologies for having dropped out of the thread, night intervened here. ;-)
>
> On 08/05/14 00:45, Ralph Castain wrote:
>
> > Okay, then we'll just have to develop a workaround for all those
> > Slurm releases where PMI-2 is borked :-(
>
> Do you know what these releases are?  Are we talking 2.6.x or 14.03?
> The 14.03 series has had a fair few rapid point releases and doesn't
> appear to be anywhere as near as stable as 2.6 was when it came out. :-(
>
> > FWIW: I think people misunderstood my statement. I specifically
> > did *not* propose to *lose* PMI-2 support. I suggested that we
> > change it to "on-by-request" instead of the current "on-by-default"
> > so we wouldn't keep getting asked about PMI-2 bugs in Slurm. Once
> > the Slurm implementation stabilized, then we could reverse that
> > policy.
> >
> > However, given that both you and Chris appear to prefer to keep it
> > "on-by-default", we'll see if we can find a way to detect that
> > PMI-2 is broken and then fall back to PMI-1.
>
> My intention was to provide the data that led us to want PMI2, but if
> configure had an option to enable PMI2 by default so that only those
> who requested it got it then I'd be more than happy - we'd just add it
> to our script to build it.
>
> All the best!
> Chris
> - --
>  Christopher SamuelSenior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
>  http://www.vlsci.org.au/  http://twitter.com/vlsci
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlNq2poACgkQO2KABBYQAh+7DwCfeahirvoQ9Wom4VNhJIIdufeP
> 7uIAnAruTnXZBn6HXhuMAlzzSsoKkXlt
> =OvH4
> -END PGP SIGNATURE-
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14733.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

That is interesting. I think I will reconstruct your experiments on my
system when I will be testing PMI selection logic. According to your
resource count numbers I can do that. I will publish my results in the list.


2014-05-08 8:51 GMT+07:00 Christopher Samuel :

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 07/05/14 18:00, Ralph Castain wrote:
>
> > Interesting - how many nodes were involved? As I said, the bad
> > scaling becomes more evident at a fairly high node count.
>
> Our x86-64 systems are low node counts (we've got BG/Q for capacity),
> the cluster that those tests were run on has 70 nodes, each with 16
> cores, so I suspect we're a long long way away from that pain point.
>
> All the best!
> Chris
> - --
>  Christopher SamuelSenior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
>  http://www.vlsci.org.au/  http://twitter.com/vlsci
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlNq4zQACgkQO2KABBYQAh8ErQCcCBFFeB5q27b7AkqfClliUdvC
> NJIAn1Cun+yY8zd6IToEsYJELpJTIdGb
> =K0XF
> -END PGP SIGNATURE-
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14734.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Artem Polyakov]

2014-05-08 9:54 GMT+07:00 Ralph Castain :

>
> On May 7, 2014, at 6:15 PM, Christopher Samuel 
> wrote:
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
> >
> > Hi all,
> >
> > Apologies for having dropped out of the thread, night intervened here.
> ;-)
> >
> > On 08/05/14 00:45, Ralph Castain wrote:
> >
> >> Okay, then we'll just have to develop a workaround for all those
> >> Slurm releases where PMI-2 is borked :-(
> >
> > Do you know what these releases are?  Are we talking 2.6.x or 14.03?
> > The 14.03 series has had a fair few rapid point releases and doesn't
> > appear to be anywhere as near as stable as 2.6 was when it came out. :-(
>
> Yeah :-(
>
> I think there was one 2.6.x that was borked, and definitely problems in
> the 14.03.x line. Can't pinpoint it for you, though.
>

The bug I experienced with abnormal OMPI termination persist starting from
2.6.3 till latest slurm release. It may appear earlier - I didn't check.
However SLURM gyus didn't confirm that it's a bug acually. Things will get
clear after 2 weeks when the person who maintains the code will review the
patch. But I am pretty sure thats a bug.

Refer to this thread
http://thread.gmane.org/gmane.comp.distributed.slurm.devel/5213.



>
> >
> >> FWIW: I think people misunderstood my statement. I specifically
> >> did *not* propose to *lose* PMI-2 support. I suggested that we
> >> change it to "on-by-request" instead of the current "on-by-default"
> >> so we wouldn't keep getting asked about PMI-2 bugs in Slurm. Once
> >> the Slurm implementation stabilized, then we could reverse that
> >> policy.
> >>
> >> However, given that both you and Chris appear to prefer to keep it
> >> "on-by-default", we'll see if we can find a way to detect that
> >> PMI-2 is broken and then fall back to PMI-1.
> >
> > My intention was to provide the data that led us to want PMI2, but if
> > configure had an option to enable PMI2 by default so that only those
> > who requested it got it then I'd be more than happy - we'd just add it
> > to our script to build it.
>
> Sounds good. I'm going to have to dig deeper into those numbers, though,
> as they don't entirely add up to me. Once the job gets launched, the launch
> method itself should have no bearing on computational speed - IF all things
> are equal. In other words, if the process layout is the same, and the
> binding pattern is the same, then computational speed should be roughly
> equivalent regardless of how the procs were started.
>
> My guess is that your data might indicate a difference in the layout
> and/or binding pattern as opposed to PMI2 vs mpirun. At the scale you
> mention later in the thread (only 70 nodes x 16 ppn), the difference in
> launch timing would be zilch. So I'm betting you would find (upon further
> exploration) that (a) you might not have been binding processes when
> launching by mpirun, since we didn't bind by default until the 1.8 series,
> but were binding under direct srun launch, and (b) your process mapping
> would quite likely be different as we default to byslot mapping, and I
> believe srun defaults to bynode?
>
> Might be worth another comparison run when someone has time.
>
>
> >
> > All the best!
> > Chris
> > - --
> > Christopher SamuelSenior Systems Administrator
> > VLSCI - Victorian Life Sciences Computation Initiative
> > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
> > http://www.vlsci.org.au/  http://twitter.com/vlsci
> >
> > -BEGIN PGP SIGNATURE-
> > Version: GnuPG v1.4.14 (GNU/Linux)
> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> > iEYEARECAAYFAlNq2poACgkQO2KABBYQAh+7DwCfeahirvoQ9Wom4VNhJIIdufeP
> > 7uIAnAruTnXZBn6HXhuMAlzzSsoKkXlt
> > =OvH4
> > -END PGP SIGNATURE-
> > ___
> > devel mailing list
> > de...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14733.php
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14738.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RFC: refactor PMI support

[Artem Polyakov]

Hi, all.

Ralph commited the code that was developed for this RFC (r31908). This
commit will brake PMI1 support. In case of hurry - apply attached patch.
Ralph will apply it once he'll be online. I have no rights for that yet.


2014-05-19 21:18 GMT+07:00 Ralph Castain :

> WHAT:Refactor the PMI support into something more flexible
>
> WHY:  We currently support both PMI-1 and PMI-2. However, a number of
> PMI-2 implementations
>(specifically, in several Slurm releases) have bugs in them
> that cause significant problems.
>In addition, we have new PMI implementations coming along
> that we would also like to support.
>The current support in OMPI is spread across multiple
> locations, each of which must track which
>PMI version is to be used. Centralizing the PMI integration
> allows us to avoid that duplication.
>
> WHERE:  https://bitbucket.org/rhc/ompi-pmi
>
> TIMEOUT:  June 3rd, after the telecon
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14827.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
Index: opal/mca/common/pmi/common_pmi.c
===
--- opal/mca/common/pmi/common_pmi.c	(revision 31908)
+++ opal/mca/common/pmi/common_pmi.c	(working copy)
@@ -59,6 +59,7 @@
  * even launched by a PMI server before attempting
  * to use PMI */
 if (NULL == getenv("PMI_FD")) {
+opal_show_help("help-common-pmi.txt", "pmi2-init-failed", true, rc);
 return OPAL_ERROR;
 }
 
@@ -124,13 +125,6 @@
 int spawned;
 int rc, ret = OPAL_ERROR;
 
-/* deal with a Slurm bug by first checking if we were
- * even launched by a PMI server before attempting
- * to use PMI */
-if (NULL == getenv("PMI_FD")) {
-return OPAL_ERROR;
-}
-
 if (PMI_SUCCESS != (rc = PMI_Initialized(&initialized))) {
 OPAL_PMI_ERROR(rc, "PMI_Initialized");
 return OPAL_ERROR;

Re: [OMPI devel] RFC: refactor PMI support

[Artem Polyakov]

Thank you, Mike!


2014-06-01 13:43 GMT+07:00 Mike Dubman :

> applied here: https://svn.open-mpi.org/trac/ompi/changeset/31909
>
>
> On Sun, Jun 1, 2014 at 9:15 AM, Artem Polyakov  wrote:
>
>> Hi, all.
>>
>> Ralph commited the code that was developed for this RFC (r31908). This
>> commit will brake PMI1 support. In case of hurry - apply attached patch.
>> Ralph will apply it once he'll be online. I have no rights for that yet.
>>
>>
>> 2014-05-19 21:18 GMT+07:00 Ralph Castain :
>>
>>> WHAT:Refactor the PMI support into something more flexible
>>>
>>> WHY:  We currently support both PMI-1 and PMI-2. However, a number
>>> of PMI-2 implementations
>>>(specifically, in several Slurm releases) have bugs in
>>> them that cause significant problems.
>>>In addition, we have new PMI implementations coming along
>>> that we would also like to support.
>>>The current support in OMPI is spread across multiple
>>> locations, each of which must track which
>>>PMI version is to be used. Centralizing the PMI
>>> integration allows us to avoid that duplication.
>>>
>>> WHERE:  https://bitbucket.org/rhc/ompi-pmi
>>>
>>> TIMEOUT:  June 3rd, after the telecon
>>>
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/05/14827.php
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Searchable archives:
>> http://www.open-mpi.org/community/lists/devel/2014/06/index.php
>>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/14920.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

[OMPI devel] OpenIB/usNIC errors

[Artem Polyakov]

Hello, while testing new PMI implementation I faced a problem with OpenIB
and/or usNIC support.
The cluster I use is build on Mellanox QDR. We don't use Cisco hardware,
thus no Cisco Virtual Interface Card. To exclude possibility of new PMI
code influence I used mpirun to launch the job. Slurm job script is
attached.

While investigating the problem I found the following:
1. With TCP btl everything works without errors (add export
OMPI_MCA_btl="tcp,self" in attached batch script).

2. With fixed OpenIB  support  (add export OMPI_MCA_btl="openib,self" in
attached batch script) I get followint error:
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
hellompi:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
mca_btl_openib_del_procs: Assertion
`((opal_object_t*)endpoint)->obj_reference_count == 1' failed.

Complete logs are tar-ed, check "openib-failure" directory.

3. If I do not fix the BTL component (no OMPI_MCA_btl is exported) I can
get either immediate fail talking about usNIC/OpenIB problems OR programs
hangs.
For both cases I'm attaching complete tar-ed logs. Check "auto-failure" dir
for ompi stdout and stderr and "auto-hang" for the hang case.

I am ready to provide additional info or help with testing but I have no
time to track the problem myself in near several days.

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov


task_mpirun.job
Description: Binary data


usnic-openib-faults.tar.bz2
Description: BZip2 compressed data

Re: [OMPI devel] OpenIB/usNIC errors

[Artem Polyakov]

P.S.

1. Just to make sure I tried the same program with old ompi-1.6.5 that is
installed on our cluster without any problem.
2. My testing program just sends data through the ring.


2014-06-01 13:57 GMT+07:00 Artem Polyakov :

> Hello, while testing new PMI implementation I faced a problem with OpenIB
> and/or usNIC support.
> The cluster I use is build on Mellanox QDR. We don't use Cisco hardware,
> thus no Cisco Virtual Interface Card. To exclude possibility of new PMI
> code influence I used mpirun to launch the job. Slurm job script is
> attached.
>
> While investigating the problem I found the following:
> 1. With TCP btl everything works without errors (add export
> OMPI_MCA_btl="tcp,self" in attached batch script).
>
> 2. With fixed OpenIB  support  (add export OMPI_MCA_btl="openib,self" in
> attached batch script) I get followint error:
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
>
> Complete logs are tar-ed, check "openib-failure" directory.
>
> 3. If I do not fix the BTL component (no OMPI_MCA_btl is exported) I can
> get either immediate fail talking about usNIC/OpenIB problems OR programs
> hangs.
> For both cases I'm attaching complete tar-ed logs. Check "auto-failure"
> dir for ompi stdout and stderr and "auto-hang" for the hang case.
>
> I am ready to provide additional info or help with testing but I have no
> time to track the problem myself in near several days.
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] OpenIB/usNIC errors

[Artem Polyakov]

I think I can do that.

воскресенье, 1 июня 2014 г. пользователь Gilles Gouaillardet написал:

> Artem,
>
> this looks like the issue initially reported by Rolf
> http://www.open-mpi.org/community/lists/devel/2014/05/14836.php
>
> in http://www.open-mpi.org/community/lists/devel/2014/05/14839.php
> i posted a patch and a workaround :
> export OMPI_MCA_btl_openib_use_eager_rdma=0
>
> i do not recall i commited the patch (Nathan is reviewing) to the trunk.
>
> if you have a chance to test it and if it works, i ll commit it tomorrow
>
> Cheers,
>
> Gilles
>
>
>
> On Sun, Jun 1, 2014 at 3:57 PM, Artem Polyakov  > wrote:
>
>>
>> 2. With fixed OpenIB  support  (add export OMPI_MCA_btl="openib,self" in
>> attached batch script) I get followint error:
>> hellompi:
>> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
>> mca_btl_openib_del_procs: Assertion
>> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
>>
>>

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] OpenIB/usNIC errors

[Artem Polyakov]

Hello, Jeff.

Please, check attached tar ("auto-failure" dir). There I've seen the
following message:
--


An internal error has occurred in the Open MPI usNIC BTL.  This is
highly unusual and shouldn't happen.  It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
  Server:   cn5

Message:  usnic connectivity client IPC connect read failed
File:
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/usnic/btl_usnic_cclient.c

  Line: 125
  Error:Operation not permitted
--

And I was wondered because as I've said we don't use Cisco hardware. My
guess that it can be a problem in query function. But I think this shows
that usnic BTL somehow participates in computiation.


2014-06-01 19:20 GMT+07:00 Jeff Squyres (jsquyres) :

> Just to be clear: it looks like you haven't seen any errors from the usnic
> BTL, right?  (the Cisco VIC uses the usnic BTL only -- it does not use the
> openib BTL)
>
>
> On Jun 1, 2014, at 2:57 AM, Artem Polyakov  wrote:
>
> > Hello, while testing new PMI implementation I faced a problem with
> OpenIB and/or usNIC support.
> > The cluster I use is build on Mellanox QDR. We don't use Cisco hardware,
> thus no Cisco Virtual Interface Card. To exclude possibility of new PMI
> code influence I used mpirun to launch the job. Slurm job script is
> attached.
> >
> > While investigating the problem I found the following:
> > 1. With TCP btl everything works without errors (add export
> OMPI_MCA_btl="tcp,self" in attached batch script).
> >
> > 2. With fixed OpenIB  support  (add export OMPI_MCA_btl="openib,self" in
> attached batch script) I get followint error:
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> > hellompi:
> /home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/openib/btl_openib.c:1151:
> mca_btl_openib_del_procs: Assertion
> `((opal_object_t*)endpoint)->obj_reference_count == 1' failed.
> >
> > Complete logs are tar-ed, check "openib-failure" directory.
> >
> > 3. If I do not fix the BTL component (no OMPI_MCA_btl is exported) I can
> get either immediate fail talking about usNIC/OpenIB problems OR programs
> hangs.
> > For both cases I'm attaching complete tar-ed logs. Check "auto-failure"
> dir for ompi stdout and stderr and "auto-hang" for the hang case.
> >
> > I am ready to provide additional info or help with testing but I have no
> time to track the problem myself in near several days.
> >
> > --
>

Re: [OMPI devel] OpenIB/usNIC errors

[Artem Polyakov]

2014-06-01 14:24 GMT+07:00 Gilles Gouaillardet <
gilles.gouaillar...@gmail.com>:

> export OMPI_MCA_btl_openib_use_eager_rdma=0


Gilles,

I test your approach. Both:
a) export OMPI_MCA_btl_openib_use_eager_rdma=0
b) applying your patch and run without "export
OMPI_MCA_btl_openib_use_eager_rdma=0"
works well for me.
This fixes first part of the problem: when OMPI_MCA_btl="openib,self"

However once I comment out this statement thus giving OMPI the right to
deside which BTL to use program hangs again. Here is additional information
that can be useful:

1. If I set 1 slot per node this problem doesn't rise.

2. If I use at least 2 cores per node I can see this hang.
Here is the backtraces for all branches of hanged program:

rank = 0
(gdb) bt
#0  0x0039522df343 in poll () from /lib64/libc.so.6
#1  0x7f1e4fb01605 in poll_dispatch (base=0x13973b0, tv=0x7fff2595ce50)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent2021/libevent/poll.c:165
#2  0x7f1e4faf601c in opal_libevent2021_event_base_loop
(base=0x13973b0, flags=3)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent2021/libevent/event.c:1631
#3  0x7f1e4fa9870a in opal_progress () at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/runtime/opal_progress.c:169
#4  0x7f1e500beb51 in ompi_mpi_init (argc=1, argv=0x7fff2595d158,
requested=0, provided=0x7fff2595cfc8)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/runtime/ompi_mpi_init.c:641
#5  0x7f1e500f425e in PMPI_Init (argc=0x7fff2595d02c,
argv=0x7fff2595d020) at pinit.c:84
#6  0x00400a6e in main ()

rank = 1
(gdb) bt
*#0  0x0039522accdd in nanosleep () from /lib64/libc.so.6*
*#1  0x0039522e1e54 in usleep () from /lib64/libc.so.6*
*> GOTCHA *
*#2  0x7fae7a6a7f4d in ompi_btl_usnic_connectivity_client_init () at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/usnic/btl_usnic_cclient.c:92*
#3  0x7fae7a6a4b72 in usnic_component_init
(num_btl_modules=0x7fffc0a67cc8, want_progress_threads=false,
want_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/usnic/btl_usnic_component.c:461
#4  0x7fae7ed9958f in mca_btl_base_select
(enable_progress_threads=false, enable_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/btl/base/btl_base_select.c:113
*< GOTCHA *
#5  0x7fae7b5e6b48 in mca_bml_r2_component_init
(priority=0x7fffc0a67d84, enable_progress_threads=false,
enable_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/bml/r2/bml_r2_component.c:88
#6  0x7fae7ed98362 in mca_bml_base_init (enable_progress_threads=false,
enable_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/bml/base/bml_base_init.c:69
#7  0x7fae79e2dcb5 in mca_pml_ob1_component_init
(priority=0x7fffc0a67eb0, enable_progress_threads=false,
enable_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/pml/ob1/pml_ob1_component.c:271
#8  0x7fae7edc0251 in mca_pml_base_select
(enable_progress_threads=false, enable_mpi_threads=false)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/mca/pml/base/pml_base_select.c:127
#9  0x7fae7ed2b9e9 in ompi_mpi_init (argc=1, argv=0x7fffc0a681c8,
requested=0, provided=0x7fffc0a68038)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/runtime/ompi_mpi_init.c:611
#10 0x7fae7ed6125e in PMPI_Init (argc=0x7fffc0a6809c,
argv=0x7fffc0a68090) at pinit.c:84
#11 0x00400a6e in main ()

rank=2
(gdb) bt
#0  0x0038e38df343 in poll () from /lib64/libc.so.6
#1  0x7fa403413605 in poll_dispatch (base=0x25e33b0, tv=0x7fff1a081be0)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent2021/libevent/poll.c:165
#2  0x7fa40340801c in opal_libevent2021_event_base_loop
(base=0x25e33b0, flags=3)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent2021/libevent/event.c:1631
#3  0x7fa4033aa70a in opal_progress () at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/runtime/opal_progress.c:169
#4  0x7fa4039d0b51 in ompi_mpi_init (argc=1, argv=0x7fff1a081ee8,
requested=0, provided=0x7fff1a081d58)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/ompi/runtime/ompi_mpi_init.c:641
#5  0x7fa403a0625e in PMPI_Init (argc=0x7fff1a081dbc,
argv=0x7fff1a081db0) at pinit.c:84
#6  0x00400a6e in main ()


rank=3
(gdb) bt
#0  0x0038e38df343 in poll () from /lib64/libc.so.6
#1  0x7f1ad8de7605 in poll_dispatch (base=0x21a73b0, tv=0x7fff0fa9f7f0)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent2021/libevent/poll.c:165
#2  0x7f1ad8ddc01c in opal_libevent2021_event_base_loop
(base=0x21a73b0, flags=3)
at
/home/research/artpol/ompi_dev//ompi-trunk_r31907/opal/mca/event/libevent

Re: [OMPI devel] RFC: refactor PMI support

[Artem Polyakov]

I did check this for SLURM 2.6.5


2014-06-01 20:31 GMT+07:00 Ralph Castain :

> That really wasn't necessary - I had tested it under PMI-1 and it was
> fine. Artem: did you test it, or just assume it wasn't right?
>
>
> On May 31, 2014, at 11:47 PM, Artem Polyakov  wrote:
>
> Thank you, Mike!
>
>
> 2014-06-01 13:43 GMT+07:00 Mike Dubman :
>
>> applied here: https://svn.open-mpi.org/trac/ompi/changeset/31909
>>
>>
>> On Sun, Jun 1, 2014 at 9:15 AM, Artem Polyakov 
>> wrote:
>>
>>> Hi, all.
>>>
>>> Ralph commited the code that was developed for this RFC (r31908). This
>>> commit will brake PMI1 support. In case of hurry - apply attached patch.
>>> Ralph will apply it once he'll be online. I have no rights for that yet.
>>>
>>>
>>> 2014-05-19 21:18 GMT+07:00 Ralph Castain :
>>>
>>>> WHAT:Refactor the PMI support into something more flexible
>>>>
>>>> WHY:  We currently support both PMI-1 and PMI-2. However, a number
>>>> of PMI-2 implementations
>>>>(specifically, in several Slurm releases) have bugs in
>>>> them that cause significant problems.
>>>>In addition, we have new PMI implementations coming
>>>> along that we would also like to support.
>>>>The current support in OMPI is spread across multiple
>>>> locations, each of which must track which
>>>>PMI version is to be used. Centralizing the PMI
>>>> integration allows us to avoid that duplication.
>>>>
>>>> WHERE:  https://bitbucket.org/rhc/ompi-pmi
>>>>
>>>> TIMEOUT:  June 3rd, after the telecon
>>>>
>>>>
>>>> ___
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/devel/2014/05/14827.php
>>>>
>>>
>>>
>>>
>>> --
>>> С Уважением, Поляков Артем Юрьевич
>>> Best regards, Artem Y. Polyakov
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Searchable archives:
>>> http://www.open-mpi.org/community/lists/devel/2014/06/index.php
>>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/06/14920.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/14921.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/14930.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

[OMPI devel] OMPI timing fix

[Artem Polyakov]

Here is quick fix of OMPI timing facility. Currently first measurement is
bogus because OMPI_PROC_MY_NAME is not initialized at the time of first
ompistart setup:

*time from start to completion of rte_init 1348381643658244 usec*
time from completion of rte_init to modex 17585 usec
time to execute modex 313168 usec
time from modex to first barrier 6565 usec
time to execute barrier 124699 usec
time from barrier to complete mpi_init 74915 usec
time to execute barrier 41990 usec

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
Index: ompi/runtime/ompi_mpi_init.c
===
--- ompi/runtime/ompi_mpi_init.c	(revision 31949)
+++ ompi/runtime/ompi_mpi_init.c	(working copy)
@@ -438,7 +438,8 @@
 mca_base_var_set_value(ret, allvalue, 4, MCA_BASE_VAR_SOURCE_DEFAULT, NULL);
 }
 
-if (ompi_enable_timing && 0 == OMPI_PROC_MY_NAME->vpid) {
+// At this moment OMPI_PROC_MY_NAME wasn't initialized yet.
+if (ompi_enable_timing /*&& 0 == OMPI_PROC_MY_NAME->vpid*/) {
 gettimeofday(&ompistart, NULL);
 }
 

Re: [OMPI devel] Agenda for next week

[Artem Polyakov]

Hello,

I would like to participate in PMI and modex discussions remotely.


2014-06-19 22:44 GMT+07:00 Jeff Squyres (jsquyres) :

> We have a bunch of topics listed on the wiki, but no real set agenda:
>
> https://svn.open-mpi.org/trac/ompi/wiki/Jun14Meeting
>
> We had remote-attendance requests for 2 topics, however, so I took the
> liberty of setting up some fixed-time webexes for them (see the wiki for
> the webex links):
>
> - Tuesday at 2pm US Central for the git discussion
> - Thursday at 9am US Central for the FT discussion
>
> Are there any other topics that people wanted to remote in to?  Fair
> warning: remote attendance is "ok" via webex, but it's no substitute for
> being there.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/15021.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] Agenda for next week

[Artem Polyakov]

It looks like we have exactly 12 hours of diff now. So I am available from
9 to 12 am your time. On what date PMI and modex conversations are
scheduled?


вторник, 24 июня 2014 г. пользователь Jeff Squyres (jsquyres) написал:

> When are you available?  We generally start at 9am US Central time.
>
>
> On Jun 19, 2014, at 9:26 PM, Artem Polyakov  > wrote:
>
> > Hello,
> >
> > I would like to participate in PMI and modex discussions remotely.
> >
> >
> > 2014-06-19 22:44 GMT+07:00 Jeff Squyres (jsquyres)  >:
> > We have a bunch of topics listed on the wiki, but no real set agenda:
> >
> > https://svn.open-mpi.org/trac/ompi/wiki/Jun14Meeting
> >
> > We had remote-attendance requests for 2 topics, however, so I took the
> liberty of setting up some fixed-time webexes for them (see the wiki for
> the webex links):
> >
> > - Tuesday at 2pm US Central for the git discussion
> > - Thursday at 9am US Central for the FT discussion
> >
> > Are there any other topics that people wanted to remote in to?  Fair
> warning: remote attendance is "ok" via webex, but it's no substitute for
> being there.
> >
> > --
> > Jeff Squyres
> > jsquy...@cisco.com 
> > For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/15021.php
> >
> >
> >
> > --
> > С Уважением, Поляков Артем Юрьевич
> > Best regards, Artem Y. Polyakov
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/15031.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com 
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/06/15044.php



-- 
Mobile mail

Re: [OMPI devel] OSHMEM out-of-date?

[Artem Polyakov]

What is it? What repository?

понедельник, 18 июля 2016 г. пользователь Ralph Castain написал:

> In file included from
> *../../../../oshmem/shmem/fortran/prototypes_shmem.h:14:0*,
>  from *../../../../oshmem/shmem/fortran/bindings.h:15*,
>  from *pshmem_put_f.c:13*:
> *pshmem_put_f.c:* In function ‘*shmem_put_f*’:
> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *warning:
> *passing argument 2 of ‘*mca_spml.spml_put*’ makes integer from pointer
> without a cast [*-Wint-conversion*]
>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
> *^*
> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
> macro ‘*FPTR_2_VOID_PTR*’
>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
> *^*
> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>  *^*
> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *note: 
> *expected
> ‘*size_t {aka long unsigned int}*’ but argument is of type ‘*void **’
>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
> *^*
> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
> macro ‘*FPTR_2_VOID_PTR*’
>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
> *^*
> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>  *^*
> In file included from *../../../../oshmem/shmem/fortran/bindings.h:16:0*,
>  from *pshmem_put_f.c:13*:
> *pshmem_put_f.c:38:25:* *warning: *passing argument 3 of ‘
> *mca_spml.spml_put*’ makes pointer from integer without a cast [
> *-Wint-conversion*]
>  OMPI_FINT_2_INT(***length),
>  *^*
> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
> definition of macro ‘*OMPI_FINT_2_INT*’
>#define OMPI_FINT_2_INT(a) *a*
>   *^*
> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>  *^*
> *pshmem_put_f.c:38:25:* *note: *expected ‘*void **’ but argument is of
> type ‘*int*’
>  OMPI_FINT_2_INT(***length),
>  *^*
> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
> definition of macro ‘*OMPI_FINT_2_INT*’
>#define OMPI_FINT_2_INT(a) *a*
>   *^*
> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>
>
>
>

-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] OSHMEM out-of-date?

[Artem Polyakov]

Ok, thank you. We will take a look

понедельник, 18 июля 2016 г. пользователь Ralph Castain написал:

> Sorry - this is on today’s master
>
> On Jul 17, 2016, at 8:31 PM, Artem Polyakov  > wrote:
>
> What is it? What repository?
>
> понедельник, 18 июля 2016 г. пользователь Ralph Castain написал:
>
>> In file included from
>> *../../../../oshmem/shmem/fortran/prototypes_shmem.h:14:0*,
>>  from *../../../../oshmem/shmem/fortran/bindings.h:15*,
>>  from *pshmem_put_f.c:13*:
>> *pshmem_put_f.c:* In function ‘*shmem_put_f*’:
>> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *warning:
>> *passing argument 2 of ‘*mca_spml.spml_put*’ makes integer from pointer
>> without a cast [*-Wint-conversion*]
>>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
>> *^*
>> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
>> macro ‘*FPTR_2_VOID_PTR*’
>>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
>> *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *note:
>> *expected ‘*size_t {aka long unsigned int}*’ but argument is of type ‘*void
>> **’
>>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
>> *^*
>> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
>> macro ‘*FPTR_2_VOID_PTR*’
>>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
>> *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> In file included from *../../../../oshmem/shmem/fortran/bindings.h:16:0*,
>>  from *pshmem_put_f.c:13*:
>> *pshmem_put_f.c:38:25:* *warning: *passing argument 3 of ‘
>> *mca_spml.spml_put*’ makes pointer from integer without a cast [
>> *-Wint-conversion*]
>>  OMPI_FINT_2_INT(***length),
>>  *^*
>> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
>> definition of macro ‘*OMPI_FINT_2_INT*’
>>#define OMPI_FINT_2_INT(a) *a*
>>   *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> *pshmem_put_f.c:38:25:* *note: *expected ‘*void **’ but argument is of
>> type ‘*int*’
>>  OMPI_FINT_2_INT(***length),
>>  *^*
>> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
>> definition of macro ‘*OMPI_FINT_2_INT*’
>>#define OMPI_FINT_2_INT(a) *a*
>>   *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>
>>
>>
>>
>
> --
> -
> Best regards, Artem Polyakov
> (Mobile mail)
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19228.php
>
>
>

-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] OSHMEM out-of-date?

[Artem Polyakov]

We have the fix. Will PR shortly.

понедельник, 18 июля 2016 г. пользователь Ralph Castain написал:

> Sorry - this is on today’s master
>
> On Jul 17, 2016, at 8:31 PM, Artem Polyakov  > wrote:
>
> What is it? What repository?
>
> понедельник, 18 июля 2016 г. пользователь Ralph Castain написал:
>
>> In file included from
>> *../../../../oshmem/shmem/fortran/prototypes_shmem.h:14:0*,
>>  from *../../../../oshmem/shmem/fortran/bindings.h:15*,
>>  from *pshmem_put_f.c:13*:
>> *pshmem_put_f.c:* In function ‘*shmem_put_f*’:
>> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *warning:
>> *passing argument 2 of ‘*mca_spml.spml_put*’ makes integer from pointer
>> without a cast [*-Wint-conversion*]
>>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
>> *^*
>> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
>> macro ‘*FPTR_2_VOID_PTR*’
>>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
>> *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> *../../../../oshmem/shmem/fortran/shmem_fortran_pointer.h:15:28:* *note:
>> *expected ‘*size_t {aka long unsigned int}*’ but argument is of type ‘*void
>> **’
>>  #define FPTR_2_VOID_PTR(a) *(*(void *)(a))
>> *^*
>> *../../../../oshmem/mca/spml/spml.h:329:44:* *note: *in expansion of
>> macro ‘*FPTR_2_VOID_PTR*’
>>  #define MCA_SPML_CALL(a) mca_spml.spml_ ## *a*
>> *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> In file included from *../../../../oshmem/shmem/fortran/bindings.h:16:0*,
>>  from *pshmem_put_f.c:13*:
>> *pshmem_put_f.c:38:25:* *warning: *passing argument 3 of ‘
>> *mca_spml.spml_put*’ makes pointer from integer without a cast [
>> *-Wint-conversion*]
>>  OMPI_FINT_2_INT(***length),
>>  *^*
>> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
>> definition of macro ‘*OMPI_FINT_2_INT*’
>>#define OMPI_FINT_2_INT(a) *a*
>>   *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>  *^*
>> *pshmem_put_f.c:38:25:* *note: *expected ‘*void **’ but argument is of
>> type ‘*int*’
>>  OMPI_FINT_2_INT(***length),
>>  *^*
>> *../../../../ompi/mpi/fortran/base/fint_2_int.h:41:30:* *note: *in
>> definition of macro ‘*OMPI_FINT_2_INT*’
>>#define OMPI_FINT_2_INT(a) *a*
>>   *^*
>> *pshmem_put_f.c:36:5:* *note: *in expansion of macro ‘*MCA_SPML_CALL*’
>>  *MCA_SPML_CALL*(put(FPTR_2_VOID_PTR(target),
>>
>>
>>
>>
>
> --
> -
> Best regards, Artem Polyakov
> (Mobile mail)
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19228.php
>
>
>

-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

Gilles, we are aware and working on this.

2016-07-21 13:53 GMT+06:00 Gilles Gouaillardet :

> Folks,
>
>
> Mellanox Jenkins marks recent PR's as failed for very surprising reasons.
>
>
> mpirun --mca btl sm,self ...
>
>
> failed because processes could not contact each other. i was able to
> reproduce this once on my workstation,
>
> and found the root cause was a dirty build and/or install dir.
>
>
> i added some debug in autogen.sh and found that :
>
> - the workspace (install dir) contains some old files
>
> - it seems all PR's use the same workspace (if it was clean, that would be
> ok as long as Jenkins process only one PR at a time)
>
> - there are currently two PR's being processed for the ompi-release repo,
> and per the log, they seem to use run from the very same directory
>
> - Jenkins for the pmix repo seems to suffer the same issue
>
>
> could someone have a look at this ?
>
>
> Cheers,
>
>
> Gilles
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19247.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

Thank you for the input by the way. It sounds very useful!

2016-07-21 13:54 GMT+06:00 Artem Polyakov :

> Gilles, we are aware and working on this.
>
> 2016-07-21 13:53 GMT+06:00 Gilles Gouaillardet :
>
>> Folks,
>>
>>
>> Mellanox Jenkins marks recent PR's as failed for very surprising reasons.
>>
>>
>> mpirun --mca btl sm,self ...
>>
>>
>> failed because processes could not contact each other. i was able to
>> reproduce this once on my workstation,
>>
>> and found the root cause was a dirty build and/or install dir.
>>
>>
>> i added some debug in autogen.sh and found that :
>>
>> - the workspace (install dir) contains some old files
>>
>> - it seems all PR's use the same workspace (if it was clean, that would
>> be ok as long as Jenkins process only one PR at a time)
>>
>> - there are currently two PR's being processed for the ompi-release repo,
>> and per the log, they seem to use run from the very same directory
>>
>> - Jenkins for the pmix repo seems to suffer the same issue
>>
>>
>> could someone have a look at this ?
>>
>>
>> Cheers,
>>
>>
>> Gilles
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2016/07/19247.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

This is fixed now. Jenkins update dropped this setting.
We dealing with some other issue now. Will update later.

четверг, 21 июля 2016 г. пользователь Jeff Squyres (jsquyres) написал:

> Gilles: Oh, sweet!  This could answer a long-standing question: why PR's
> sometimes fail with unexplained Libtool / depcomp problems.
>
> Artem: I'm mailing list several hours after your initial exchange with
> Gilles, so you may have solved this by now, but since your Jenkins was
> running multiple Open MPI builds simultaneously in the same workspace, that
> could definitely lead to the Libtool / depcomp issues that we have talked
> about previously (and potentially other issues).  If you haven't done so
> already, you will likely need to make sure that simultaneous OMPI Jenkins
> builds are totally divorced from each other: different source dir,
> different build dir, different install dir.
>
>
>
>
> > On Jul 21, 2016, at 3:56 AM, Artem Polyakov  > wrote:
> >
> > Thank you for the input by the way. It sounds very useful!
> >
> > 2016-07-21 13:54 GMT+06:00 Artem Polyakov  >:
> > Gilles, we are aware and working on this.
> >
> > 2016-07-21 13:53 GMT+06:00 Gilles Gouaillardet  >:
> > Folks,
> >
> >
> > Mellanox Jenkins marks recent PR's as failed for very surprising reasons.
> >
> >
> > mpirun --mca btl sm,self ...
> >
> >
> > failed because processes could not contact each other. i was able to
> reproduce this once on my workstation,
> >
> > and found the root cause was a dirty build and/or install dir.
> >
> >
> > i added some debug in autogen.sh and found that :
> >
> > - the workspace (install dir) contains some old files
> >
> > - it seems all PR's use the same workspace (if it was clean, that would
> be ok as long as Jenkins process only one PR at a time)
> >
> > - there are currently two PR's being processed for the ompi-release
> repo, and per the log, they seem to use run from the very same directory
> >
> > - Jenkins for the pmix repo seems to suffer the same issue
> >
> >
> > could someone have a look at this ?
> >
> >
> > Cheers,
> >
> >
> > Gilles
> >
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19247.php
> >
> >
> >
> > --
> > С Уважением, Поляков Артем Юрьевич
> > Best regards, Artem Y. Polyakov
> >
> >
> >
> > --
> > С Уважением, Поляков Артем Юрьевич
> > Best regards, Artem Y. Polyakov
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19249.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com 
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19251.php



-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

We see the following error:

*14:26:55* + taskset -c 2,3 timeout -s SIGSEGV 15m
/var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun
-np 8 -bind-to none -mca pml ob1 -mca btl self,tcp taskset -c 2,3
/var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/examples/hello_c*14:26:55*
/var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun:
Error: unknown option "-np"


The reason is that /lib has no "openmpi" directory. My guess that
it is somehow removed during jenkins execution.

I'm checking now.

2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :

> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
> wrote:
> >
> > Folks,
> >
> > Mellanox Jenkins marks recent PR's as failed for very surprising reasons.
> >
> > mpirun --mca btl sm,self ...
> >
> > failed because processes could not contact each other. i was able to
> reproduce this once on my workstation,
> >
> > and found the root cause was a dirty build and/or install dir.
>
> Gilles: I note that these tests also failed in MTT last night (e.g., at
> Absoft, which *rarely* has errors):
>
> https://mtt.open-mpi.org/index.php?do_redir=2336
>
> Are you sure that the cause is a dirty build and/or install dir?
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

Ha!

funny that we chose exactly your PR to test jenkins :)

2016-07-21 20:26 GMT+06:00 Gilles Gouaillardet <
gilles.gouaillar...@gmail.com>:

> I explicitly removed this directory in autogen.sh of
> https://github.com/open-mpi/ompi/pull/1891
>
> if only this pr is causing this error, then please disregard it until I
> update it tomorrow.
>
> note this log suggests a workspace shared by all pr, so I guess this is
> obsolete now
>
> Cheers,
>
> Gilles
>
>
>
> On Thursday, July 21, 2016, Artem Polyakov  wrote:
>
>> We see the following error:
>>
>> *14:26:55* + taskset -c 2,3 timeout -s SIGSEGV 15m 
>> /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun 
>> -np 8 -bind-to none -mca pml ob1 -mca btl self,tcp taskset -c 2,3 
>> /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/examples/hello_c*14:26:55*
>>  /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun: 
>> Error: unknown option "-np"
>>
>>
>> The reason is that /lib has no "openmpi" directory. My guess
>> that it is somehow removed during jenkins execution.
>>
>> I'm checking now.
>>
>> 2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :
>>
>>> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
>>> wrote:
>>> >
>>> > Folks,
>>> >
>>> > Mellanox Jenkins marks recent PR's as failed for very surprising
>>> reasons.
>>> >
>>> > mpirun --mca btl sm,self ...
>>> >
>>> > failed because processes could not contact each other. i was able to
>>> reproduce this once on my workstation,
>>> >
>>> > and found the root cause was a dirty build and/or install dir.
>>>
>>> Gilles: I note that these tests also failed in MTT last night (e.g., at
>>> Absoft, which *rarely* has errors):
>>>
>>> https://mtt.open-mpi.org/index.php?do_redir=2336
>>>
>>> Are you sure that the cause is a dirty build and/or install dir?
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19256.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

We run autogen.pl when doin'g make_tarball this breaks jenkins.

2016-07-21 20:26 GMT+06:00 Gilles Gouaillardet <
gilles.gouaillar...@gmail.com>:

> I explicitly removed this directory in autogen.sh of
> https://github.com/open-mpi/ompi/pull/1891
>
> if only this pr is causing this error, then please disregard it until I
> update it tomorrow.
>
> note this log suggests a workspace shared by all pr, so I guess this is
> obsolete now
>
> Cheers,
>
> Gilles
>
>
>
> On Thursday, July 21, 2016, Artem Polyakov  wrote:
>
>> We see the following error:
>>
>> *14:26:55* + taskset -c 2,3 timeout -s SIGSEGV 15m 
>> /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun 
>> -np 8 -bind-to none -mca pml ob1 -mca btl self,tcp taskset -c 2,3 
>> /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/examples/hello_c*14:26:55*
>>  /var/lib/jenkins/jobs/gh-ompi-master-pr/workspace/ompi_install1/bin/mpirun: 
>> Error: unknown option "-np"
>>
>>
>> The reason is that /lib has no "openmpi" directory. My guess
>> that it is somehow removed during jenkins execution.
>>
>> I'm checking now.
>>
>> 2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :
>>
>>> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
>>> wrote:
>>> >
>>> > Folks,
>>> >
>>> > Mellanox Jenkins marks recent PR's as failed for very surprising
>>> reasons.
>>> >
>>> > mpirun --mca btl sm,self ...
>>> >
>>> > failed because processes could not contact each other. i was able to
>>> reproduce this once on my workstation,
>>> >
>>> > and found the root cause was a dirty build and/or install dir.
>>>
>>> Gilles: I note that these tests also failed in MTT last night (e.g., at
>>> Absoft, which *rarely* has errors):
>>>
>>> https://mtt.open-mpi.org/index.php?do_redir=2336
>>>
>>> Are you sure that the cause is a dirty build and/or install dir?
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19256.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

I see the same error with `sm,self` and `vader,self` in the PR
https://github.com/open-mpi/ompi/pull/1883.

`openib` and `tcp` works fine. Seems like regression.

2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :

> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
> wrote:
> >
> > Folks,
> >
> > Mellanox Jenkins marks recent PR's as failed for very surprising reasons.
> >
> > mpirun --mca btl sm,self ...
> >
> > failed because processes could not contact each other. i was able to
> reproduce this once on my workstation,
> >
> > and found the root cause was a dirty build and/or install dir.
>
> Gilles: I note that these tests also failed in MTT last night (e.g., at
> Absoft, which *rarely* has errors):
>
> https://mtt.open-mpi.org/index.php?do_redir=2336
>
> Are you sure that the cause is a dirty build and/or install dir?
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

Yes I though so as well. I see that only 2 checks was passed when your PR
was merged so it might be.

2016-07-21 21:23 GMT+06:00 Ralph Castain :

> I’m checking this - could be something to do with the recent PMIx update
>
> On Jul 21, 2016, at 8:21 AM, Artem Polyakov  wrote:
>
> I see the same error with `sm,self` and `vader,self` in the PR
> https://github.com/open-mpi/ompi/pull/1883.
>
> `openib` and `tcp` works fine. Seems like regression.
>
> 2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :
>
>> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
>> wrote:
>> >
>> > Folks,
>> >
>> > Mellanox Jenkins marks recent PR's as failed for very surprising
>> reasons.
>> >
>> > mpirun --mca btl sm,self ...
>> >
>> > failed because processes could not contact each other. i was able to
>> reproduce this once on my workstation,
>> >
>> > and found the root cause was a dirty build and/or install dir.
>>
>> Gilles: I note that these tests also failed in MTT last night (e.g., at
>> Absoft, which *rarely* has errors):
>>
>> https://mtt.open-mpi.org/index.php?do_redir=2336
>>
>> Are you sure that the cause is a dirty build and/or install dir?
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19260.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19261.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

correction: 3 out of 5 checks passed.

2016-07-21 21:24 GMT+06:00 Artem Polyakov :

> Yes I though so as well. I see that only 2 checks was passed when your PR
> was merged so it might be.
>
> 2016-07-21 21:23 GMT+06:00 Ralph Castain :
>
>> I’m checking this - could be something to do with the recent PMIx update
>>
>> On Jul 21, 2016, at 8:21 AM, Artem Polyakov  wrote:
>>
>> I see the same error with `sm,self` and `vader,self` in the PR
>> https://github.com/open-mpi/ompi/pull/1883.
>>
>> `openib` and `tcp` works fine. Seems like regression.
>>
>> 2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) :
>>
>>> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
>>> wrote:
>>> >
>>> > Folks,
>>> >
>>> > Mellanox Jenkins marks recent PR's as failed for very surprising
>>> reasons.
>>> >
>>> > mpirun --mca btl sm,self ...
>>> >
>>> > failed because processes could not contact each other. i was able to
>>> reproduce this once on my workstation,
>>> >
>>> > and found the root cause was a dirty build and/or install dir.
>>>
>>> Gilles: I note that these tests also failed in MTT last night (e.g., at
>>> Absoft, which *rarely* has errors):
>>>
>>> https://mtt.open-mpi.org/index.php?do_redir=2336
>>>
>>> Are you sure that the cause is a dirty build and/or install dir?
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2016/07/19260.php
>>
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2016/07/19261.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] about Mellanox Jenkins

[Artem Polyakov]

Thank you!

2016-07-21 22:05 GMT+06:00 Ralph Castain :

> I’ve got this fixed in PR https://github.com/open-mpi/ompi/pull/1897
>
>
> > On Jul 21, 2016, at 8:31 AM, Jeff Squyres (jsquyres) 
> wrote:
> >
> > FWIW, we have the Travis issue solved on master (see
> https://github.com/open-mpi/ompi/commit/af23dcc1239188e06c1b71f0735a83edc45178f2
> if you care).  I just filed a v2.x PR to get the fix over there, too.
> >
> > However, it looks like Travis doesn't merge to current HEAD when it's
> doing building, so existing PRs -- if they are not rebased -- won't see the
> Travis fix.
> >
> >
> >> On Jul 21, 2016, at 11:28 AM, Ralph Castain  wrote:
> >>
> >> Yeah - Travis was dead for the issues cited elsewhere, and Mellanox
> failed for other reasons (thread-related, distclean, or some such as I
> recall). I’m checking the builds now - suspect it has to do with the new
> PMIx_Get retrieval rules
> >>
> >>
> >>> On Jul 21, 2016, at 8:25 AM, Artem Polyakov 
> wrote:
> >>>
> >>> correction: 3 out of 5 checks passed.
> >>>
> >>> 2016-07-21 21:24 GMT+06:00 Artem Polyakov :
> >>> Yes I though so as well. I see that only 2 checks was passed when your
> PR was merged so it might be.
> >>>
> >>> 2016-07-21 21:23 GMT+06:00 Ralph Castain :
> >>> I’m checking this - could be something to do with the recent PMIx
> update
> >>>
> >>>> On Jul 21, 2016, at 8:21 AM, Artem Polyakov 
> wrote:
> >>>>
> >>>> I see the same error with `sm,self` and `vader,self` in the PR
> https://github.com/open-mpi/ompi/pull/1883.
> >>>>
> >>>> `openib` and `tcp` works fine. Seems like regression.
> >>>>
> >>>> 2016-07-21 20:11 GMT+06:00 Jeff Squyres (jsquyres) <
> jsquy...@cisco.com>:
> >>>> On Jul 21, 2016, at 3:53 AM, Gilles Gouaillardet 
> wrote:
> >>>>>
> >>>>> Folks,
> >>>>>
> >>>>> Mellanox Jenkins marks recent PR's as failed for very surprising
> reasons.
> >>>>>
> >>>>> mpirun --mca btl sm,self ...
> >>>>>
> >>>>> failed because processes could not contact each other. i was able to
> reproduce this once on my workstation,
> >>>>>
> >>>>> and found the root cause was a dirty build and/or install dir.
> >>>>
> >>>> Gilles: I note that these tests also failed in MTT last night (e.g.,
> at Absoft, which *rarely* has errors):
> >>>>
> >>>>https://mtt.open-mpi.org/index.php?do_redir=2336
> >>>>
> >>>> Are you sure that the cause is a dirty build and/or install dir?
> >>>>
> >>>> --
> >>>> Jeff Squyres
> >>>> jsquy...@cisco.com
> >>>> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >>>>
> >>>> ___
> >>>> devel mailing list
> >>>> de...@open-mpi.org
> >>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19254.php
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> С Уважением, Поляков Артем Юрьевич
> >>>> Best regards, Artem Y. Polyakov
> >>>> ___
> >>>> devel mailing list
> >>>> de...@open-mpi.org
> >>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19260.php
> >>>
> >>>
> >>> ___
> >>> devel mailing list
> >>> de...@open-mpi.org
> >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19261.php
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>>
> >>>
> >>>
> >>> --
> >>> С Уважением, Поляков Артем Юрьевич
> >>> Best regards, Artem Y. Polyakov
> >>> ___
> >>> devel mailing list
> >>> de...@open-mpi.org
> >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19263.php
> >>
> >> ___
> >> devel mailing list
> >> de...@open-mpi.org
> >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19264.php
> >
> >
> > --
> > Jeff Squyres
> > jsquy...@cisco.com
> > For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >
> > ___
> > devel mailing list
> > de...@open-mpi.org
> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19265.php
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19269.php




-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

Hi, Christoph

Please, check https://github.com/open-mpi/ompi/wiki/Request-refactoring-test
for the testing methodology and
https://github.com/open-mpi/2016-summer-perf-testing
for examples and launch scripts.

2016-08-23 21:20 GMT+07:00 Christoph Niethammer :

> Hello,
>
> I just came over this and would like to contribute some results from our
> system(s).
> Are there any specific configure options you want to see enabled beside
> --enable-mpi-thread-multiple?
> How to commit results?
>
> Best
> Christoph Niethammer
>
>
>
> - Original Message -
> From: "Arm Patinyasakdikul (apatinya)" 
> To: "Open MPI Developers" 
> Sent: Friday, July 29, 2016 8:41:06 PM
> Subject: Re: [OMPI devel] Performance analysis proposal
>
> Hey Artem, all,
>
> Thank you for the benchmark prototype. I have created the discussion page
> here : https://github.com/open-mpi/2016-summer-perf-testing/issues/1 .
>
>
> * There, I have single threaded and multithreaded performance posted.
> * The script I used is now in the repo (also in the discussion page)
> * Result with openib will come up probably next week. I have to access
> UTK machine for that.
> * I did some test and yes, I have seen some openib hang in
> multithreaded case.
> Thank you,
> Arm
>
> From: devel < devel-boun...@lists.open-mpi.org > on behalf of Artem
> Polyakov < artpo...@gmail.com >
> Reply-To: Open MPI Developers < devel@lists.open-mpi.org >
> Date: Thursday, July 28, 2016 at 10:42 PM
> To: Open MPI Developers < devel@lists.open-mpi.org >
> Subject: Re: [OMPI devel] Performance analysis proposal
>
> Thank you, Arm!
>
> Good to have vader results (I haven't tried it myself yet). Few
> comments/questions:
> 1. I guess we also want to have 1-threaded performance for the "baseline"
> reference.
> 2. Have you tried to run with openib, as I mentioned on the call I had
> some problems with it and I'm curious if you can reproduce in your
> environment.
>
> Github issue sounds good for me!
>
> 2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) <
> apati...@cisco.com > :
>
>
> I added some result to https://github.com/open-mpi/
> 2016-summer-perf-testing
>
> The result shows much better performance from 2.0.0 and master over 1.10.3
> for vader. The test ran with Artem’s version of benchmark on OB1, single
> node, bind to socket.
>
> We should have a place to discuss/comment/collaborate on results. Should I
> open an issue on that repo? So we can have github style
> commenting/referencing.
>
>
> Arm
>
>
>
>
> On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" <
> devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com > wrote:
>
> >On Jul 28, 2016, at 6:28 AM, Artem Polyakov < artpo...@gmail.com > wrote:
> >>
> >> Jeff and others,
> >>
> >> 1. The benchmark was updated to support shared memory case.
> >> 2. The wiki was updated with the benchmark description:
> https://github.com/open-mpi/ompi/wiki/Request-refactoring-
> test#benchmark-prototype
> >
> >Sweet -- thanks!
> >
> >> Let me know if we want to put this prototype to some general place. I
> think it makes sense.
> >
> >I just created:
> >
> > https://github.com/open-mpi/2016-summer-perf-testing
> >
> >Want to put it there?
> >
> >Arm just ran a bunch of tests today and will be committing a bunch of
> results in there shortly.
> >
> >--
> >Jeff Squyres
> > jsquy...@cisco.com
> >For corporate legal information go to: http://www.cisco.com/web/
> about/doing_business/legal/cri/
> >
> >___
> >devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

That's a good question. I have results myself and I don't know where to
place them.
I think that Arm's repo is not a right place to collect the data.

Jeff, can we create the repo in open mpi organization on github or do we
have something appropriate already?

четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:

> Hi Artem,
>
> Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader performance
> regression patch under
> https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc9c0847b3
> 091601919f.patch
> and master. I will do the 2.0.1rc in the next days as well.
>
> Is it possible to add me to the results repository at github or should I
> fork and request you to pull?
>
> Best
> Christoph
>
>
> - Original Message -
> From: "Artem Polyakov" >
> To: "Open MPI Developers" >
> Sent: Tuesday, August 23, 2016 5:13:30 PM
> Subject: Re: [OMPI devel] Performance analysis proposal
>
> Hi, Christoph
>
> Please, check https://github.com/open-mpi/ompi/wiki/Request-refactoring-
> test for the testing methodology and
> https://github.com/open-mpi/2016-summer-perf-testing
> for examples and launch scripts.
>
> 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de
>  > :
>
>
> Hello,
>
> I just came over this and would like to contribute some results from our
> system(s).
> Are there any specific configure options you want to see enabled beside
> --enable-mpi-thread-multiple?
> How to commit results?
>
> Best
> Christoph Niethammer
>
>
>
> - Original Message -
> From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com 
> >
> To: "Open MPI Developers" < devel@lists.open-mpi.org  >
> Sent: Friday, July 29, 2016 8:41:06 PM
> Subject: Re: [OMPI devel] Performance analysis proposal
>
> Hey Artem, all,
>
> Thank you for the benchmark prototype. I have created the discussion page
> here : https://github.com/open-mpi/2016-summer-perf-testing/issues/1 .
>
>
> * There, I have single threaded and multithreaded performance posted.
> * The script I used is now in the repo (also in the discussion page)
> * Result with openib will come up probably next week. I have to access UTK
> machine for that.
> * I did some test and yes, I have seen some openib hang in multithreaded
> case.
> Thank you,
> Arm
>
> From: devel < devel-boun...@lists.open-mpi.org  > on behalf
> of Artem Polyakov < artpo...@gmail.com  >
> Reply-To: Open MPI Developers < devel@lists.open-mpi.org  >
> Date: Thursday, July 28, 2016 at 10:42 PM
> To: Open MPI Developers < devel@lists.open-mpi.org  >
> Subject: Re: [OMPI devel] Performance analysis proposal
>
> Thank you, Arm!
>
> Good to have vader results (I haven't tried it myself yet). Few
> comments/questions:
> 1. I guess we also want to have 1-threaded performance for the "baseline"
> reference.
> 2. Have you tried to run with openib, as I mentioned on the call I had
> some problems with it and I'm curious if you can reproduce in your
> environment.
>
> Github issue sounds good for me!
>
> 2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) <
> apati...@cisco.com  > :
>
>
> I added some result to https://github.com/open-mpi/
> 2016-summer-perf-testing
>
> The result shows much better performance from 2.0.0 and master over 1.10.3
> for vader. The test ran with Artem’s version of benchmark on OB1, single
> node, bind to socket.
>
> We should have a place to discuss/comment/collaborate on results. Should I
> open an issue on that repo? So we can have github style
> commenting/referencing.
>
>
> Arm
>
>
>
>
> On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" <
> devel-boun...@lists.open-mpi.org  on behalf of
> jsquy...@cisco.com  > wrote:
>
> >On Jul 28, 2016, at 6:28 AM, Artem Polyakov < artpo...@gmail.com
>  > wrote:
> >>
> >> Jeff and others,
> >>
> >> 1. The benchmark was updated to support shared memory case.
> >> 2. The wiki was updated with the benchmark description:
> https://github.com/open-mpi/ompi/wiki/Request-refactoring-
> test#benchmark-prototype
> >
> >Sweet -- thanks!
> >
> >> Let me know if we want to put this prototype to some general place. I
> think it makes sense.
> >
> >I just created:
> >
> > https://github.com/open-mpi/2016-summer-perf-testing
> >
> >Want to put it there?
> >
> >Arm just ran a bunch of tests today and will be committing a bunch of
> results in there shortly.
> >
> >

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

If we are serious about this problem I don't see why we can't create a repo
for this data and keep the history of all measurements.

Is there any chance that we will not came up with well defined set of tests
and drop the ball here?

пятница, 26 августа 2016 г. пользователь George Bosilca написал:

> Arm repo is a good location until we converge to a well-defined set of
> tests.
>
>   George.
>
>
> On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov  > wrote:
>
>> That's a good question. I have results myself and I don't know where to
>> place them.
>> I think that Arm's repo is not a right place to collect the data.
>>
>> Jeff, can we create the repo in open mpi organization on github or do we
>> have something appropriate already?
>>
>> четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:
>>
>> Hi Artem,
>>>
>>> Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader
>>> performance regression patch under
>>> https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc
>>> 9c0847b3091601919f.patch
>>> and master. I will do the 2.0.1rc in the next days as well.
>>>
>>> Is it possible to add me to the results repository at github or should I
>>> fork and request you to pull?
>>>
>>> Best
>>> Christoph
>>>
>>>
>>> - Original Message -
>>> From: "Artem Polyakov" 
>>> To: "Open MPI Developers" 
>>> Sent: Tuesday, August 23, 2016 5:13:30 PM
>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>
>>> Hi, Christoph
>>>
>>> Please, check https://github.com/open-mpi/om
>>> pi/wiki/Request-refactoring-test for the testing methodology and
>>> https://github.com/open-mpi/2016-summer-perf-testing
>>> for examples and launch scripts.
>>>
>>> 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de > :
>>>
>>>
>>> Hello,
>>>
>>> I just came over this and would like to contribute some results from our
>>> system(s).
>>> Are there any specific configure options you want to see enabled beside
>>> --enable-mpi-thread-multiple?
>>> How to commit results?
>>>
>>> Best
>>> Christoph Niethammer
>>>
>>>
>>>
>>> - Original Message -
>>> From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com >
>>> To: "Open MPI Developers" < devel@lists.open-mpi.org >
>>> Sent: Friday, July 29, 2016 8:41:06 PM
>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>
>>> Hey Artem, all,
>>>
>>> Thank you for the benchmark prototype. I have created the discussion
>>> page here : https://github.com/open-mpi/2016-summer-perf-testing/issues/
>>> 1 .
>>>
>>>
>>> * There, I have single threaded and multithreaded performance posted.
>>> * The script I used is now in the repo (also in the discussion page)
>>> * Result with openib will come up probably next week. I have to access
>>> UTK machine for that.
>>> * I did some test and yes, I have seen some openib hang in multithreaded
>>> case.
>>> Thank you,
>>> Arm
>>>
>>> From: devel < devel-boun...@lists.open-mpi.org > on behalf of Artem
>>> Polyakov < artpo...@gmail.com >
>>> Reply-To: Open MPI Developers < devel@lists.open-mpi.org >
>>> Date: Thursday, July 28, 2016 at 10:42 PM
>>> To: Open MPI Developers < devel@lists.open-mpi.org >
>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>
>>> Thank you, Arm!
>>>
>>> Good to have vader results (I haven't tried it myself yet). Few
>>> comments/questions:
>>> 1. I guess we also want to have 1-threaded performance for the
>>> "baseline" reference.
>>> 2. Have you tried to run with openib, as I mentioned on the call I had
>>> some problems with it and I'm curious if you can reproduce in your
>>> environment.
>>>
>>> Github issue sounds good for me!
>>>
>>> 2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) <
>>> apati...@cisco.com > :
>>>
>>>
>>> I added some result to https://github.com/open-mpi/20
>>> 16-summer-perf-testing
>>>
>>> The result shows much better performance from 2.0.0 and master over
>>> 1.10.3 for vader. The test ran wit

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

This is a good idea. We also have data with SM/Vader to discuss. I'll send
them later this week.

Do you think of regular calls or per agreement?

пятница, 26 августа 2016 г. пользователь George Bosilca написал:

> We are serious about this. However, we not only have to define a set of
> meaningful tests (which we don't have yet) but also decide the conditions
> in which they are executed, and more critically what additional information
> we need to make them reproducible, understandable and comparable.
>
> We started discussion on these topics during the developers meeting few
> weeks ago, but we barely define what we think will be necessary for trivial
> tests such as single threaded bandwidth. It might be worth having a regular
> phone call (in addition to the Tuesday morning) to make progress.
>
>   George.
>
>
> On Thu, Aug 25, 2016 at 9:37 PM, Artem Polyakov  > wrote:
>
>> If we are serious about this problem I don't see why we can't create a
>> repo for this data and keep the history of all measurements.
>>
>> Is there any chance that we will not came up with well defined set of
>> tests and drop the ball here?
>>
>> пятница, 26 августа 2016 г. пользователь George Bosilca написал:
>>
>> Arm repo is a good location until we converge to a well-defined set of
>>> tests.
>>>
>>>   George.
>>>
>>>
>>> On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov 
>>> wrote:
>>>
>>>> That's a good question. I have results myself and I don't know where to
>>>> place them.
>>>> I think that Arm's repo is not a right place to collect the data.
>>>>
>>>> Jeff, can we create the repo in open mpi organization on github or do
>>>> we have something appropriate already?
>>>>
>>>> четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:
>>>>
>>>> Hi Artem,
>>>>>
>>>>> Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader
>>>>> performance regression patch under
>>>>> https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc
>>>>> 9c0847b3091601919f.patch
>>>>> and master. I will do the 2.0.1rc in the next days as well.
>>>>>
>>>>> Is it possible to add me to the results repository at github or should
>>>>> I fork and request you to pull?
>>>>>
>>>>> Best
>>>>> Christoph
>>>>>
>>>>>
>>>>> - Original Message -
>>>>> From: "Artem Polyakov" 
>>>>> To: "Open MPI Developers" 
>>>>> Sent: Tuesday, August 23, 2016 5:13:30 PM
>>>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>>>
>>>>> Hi, Christoph
>>>>>
>>>>> Please, check https://github.com/open-mpi/om
>>>>> pi/wiki/Request-refactoring-test for the testing methodology and
>>>>> https://github.com/open-mpi/2016-summer-perf-testing
>>>>> for examples and launch scripts.
>>>>>
>>>>> 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de
>>>>> > :
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> I just came over this and would like to contribute some results from
>>>>> our system(s).
>>>>> Are there any specific configure options you want to see enabled
>>>>> beside --enable-mpi-thread-multiple?
>>>>> How to commit results?
>>>>>
>>>>> Best
>>>>> Christoph Niethammer
>>>>>
>>>>>
>>>>>
>>>>> - Original Message -
>>>>> From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com >
>>>>> To: "Open MPI Developers" < devel@lists.open-mpi.org >
>>>>> Sent: Friday, July 29, 2016 8:41:06 PM
>>>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>>>
>>>>> Hey Artem, all,
>>>>>
>>>>> Thank you for the benchmark prototype. I have created the discussion
>>>>> page here : https://github.com/open-mpi/20
>>>>> 16-summer-perf-testing/issues/1 .
>>>>>
>>>>>
>>>>> * There, I have single threaded and multithreaded performance posted.
>>>>> * The script I used is now in the repo (also in the dis

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

I've marked the first week.

2016-08-26 19:26 GMT+07:00 George Bosilca :

> Let's go regular for a period and then adapt.
>
> For everybody interested in the performance discussion, I setup a doodle
> for next week. The dates themselves are not important, we need a regular
> timeslot. Please answer with the idea that we do 4 weeks in a row and then
> assess the situation and dece if we need to continue, decrease the
> frequency or declare the problem solved. Here is the participation link:
> http://doodle.com/poll/w4fkb9gr3h2q5p6v
>
>   George.
>
>
> On Fri, Aug 26, 2016 at 6:39 AM, Artem Polyakov 
> wrote:
>
>> This is a good idea. We also have data with SM/Vader to discuss. I'll
>> send them later this week.
>>
>> Do you think of regular calls or per agreement?
>>
>> пятница, 26 августа 2016 г. пользователь George Bosilca написал:
>>
>>> We are serious about this. However, we not only have to define a set of
>>> meaningful tests (which we don't have yet) but also decide the conditions
>>> in which they are executed, and more critically what additional information
>>> we need to make them reproducible, understandable and comparable.
>>>
>>> We started discussion on these topics during the developers meeting few
>>> weeks ago, but we barely define what we think will be necessary for trivial
>>> tests such as single threaded bandwidth. It might be worth having a regular
>>> phone call (in addition to the Tuesday morning) to make progress.
>>>
>>>   George.
>>>
>>>
>>> On Thu, Aug 25, 2016 at 9:37 PM, Artem Polyakov 
>>> wrote:
>>>
>>>> If we are serious about this problem I don't see why we can't create a
>>>> repo for this data and keep the history of all measurements.
>>>>
>>>> Is there any chance that we will not came up with well defined set of
>>>> tests and drop the ball here?
>>>>
>>>> пятница, 26 августа 2016 г. пользователь George Bosilca написал:
>>>>
>>>> Arm repo is a good location until we converge to a well-defined set of
>>>>> tests.
>>>>>
>>>>>   George.
>>>>>
>>>>>
>>>>> On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov 
>>>>> wrote:
>>>>>
>>>>>> That's a good question. I have results myself and I don't know where
>>>>>> to place them.
>>>>>> I think that Arm's repo is not a right place to collect the data.
>>>>>>
>>>>>> Jeff, can we create the repo in open mpi organization on github or do
>>>>>> we have something appropriate already?
>>>>>>
>>>>>> четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:
>>>>>>
>>>>>> Hi Artem,
>>>>>>>
>>>>>>> Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader
>>>>>>> performance regression patch under
>>>>>>> https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc
>>>>>>> 9c0847b3091601919f.patch
>>>>>>> and master. I will do the 2.0.1rc in the next days as well.
>>>>>>>
>>>>>>> Is it possible to add me to the results repository at github or
>>>>>>> should I fork and request you to pull?
>>>>>>>
>>>>>>> Best
>>>>>>> Christoph
>>>>>>>
>>>>>>>
>>>>>>> - Original Message -
>>>>>>> From: "Artem Polyakov" 
>>>>>>> To: "Open MPI Developers" 
>>>>>>> Sent: Tuesday, August 23, 2016 5:13:30 PM
>>>>>>> Subject: Re: [OMPI devel] Performance analysis proposal
>>>>>>>
>>>>>>> Hi, Christoph
>>>>>>>
>>>>>>> Please, check https://github.com/open-mpi/om
>>>>>>> pi/wiki/Request-refactoring-test for the testing methodology and
>>>>>>> https://github.com/open-mpi/2016-summer-perf-testing
>>>>>>> for examples and launch scripts.
>>>>>>>
>>>>>>> 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de
>>>>>>> > :
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>&g

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

I'd prefer it to be created.

2016-08-26 20:59 GMT+07:00 Jeff Squyres (jsquyres) :

> Sorry for jumping in so late.
>
> Honestly, there's no problem with making a repo in the open-mpi github
> org.  It's just as trivial to make one there as anywhere else.
>
> Let me know if you want one.
>
>
> > On Aug 26, 2016, at 8:46 AM, Artem Polyakov  wrote:
> >
> > I've marked the first week.
> >
> > 2016-08-26 19:26 GMT+07:00 George Bosilca :
> > Let's go regular for a period and then adapt.
> >
> > For everybody interested in the performance discussion, I setup a doodle
> for next week. The dates themselves are not important, we need a regular
> timeslot. Please answer with the idea that we do 4 weeks in a row and then
> assess the situation and dece if we need to continue, decrease the
> frequency or declare the problem solved. Here is the participation link:
> http://doodle.com/poll/w4fkb9gr3h2q5p6v
> >
> >   George.
> >
> >
> > On Fri, Aug 26, 2016 at 6:39 AM, Artem Polyakov 
> wrote:
> > This is a good idea. We also have data with SM/Vader to discuss. I'll
> send them later this week.
> >
> > Do you think of regular calls or per agreement?
> >
> > пятница, 26 августа 2016 г. пользователь George Bosilca написал:
> > We are serious about this. However, we not only have to define a set of
> meaningful tests (which we don't have yet) but also decide the conditions
> in which they are executed, and more critically what additional information
> we need to make them reproducible, understandable and comparable.
> >
> > We started discussion on these topics during the developers meeting few
> weeks ago, but we barely define what we think will be necessary for trivial
> tests such as single threaded bandwidth. It might be worth having a regular
> phone call (in addition to the Tuesday morning) to make progress.
> >
> >   George.
> >
> >
> > On Thu, Aug 25, 2016 at 9:37 PM, Artem Polyakov 
> wrote:
> > If we are serious about this problem I don't see why we can't create a
> repo for this data and keep the history of all measurements.
> >
> > Is there any chance that we will not came up with well defined set of
> tests and drop the ball here?
> >
> > пятница, 26 августа 2016 г. пользователь George Bosilca написал:
> >
> > Arm repo is a good location until we converge to a well-defined set of
> tests.
> >
> >   George.
> >
> >
> > On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov 
> wrote:
> > That's a good question. I have results myself and I don't know where to
> place them.
> > I think that Arm's repo is not a right place to collect the data.
> >
> > Jeff, can we create the repo in open mpi organization on github or do we
> have something appropriate already?
> >
> > четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:
> >
> > Hi Artem,
> >
> > Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader
> performance regression patch under
> > https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc9c0847b3
> 091601919f.patch
> > and master. I will do the 2.0.1rc in the next days as well.
> >
> > Is it possible to add me to the results repository at github or should I
> fork and request you to pull?
> >
> > Best
> > Christoph
> >
> >
> > - Original Message -
> > From: "Artem Polyakov" 
> > To: "Open MPI Developers" 
> > Sent: Tuesday, August 23, 2016 5:13:30 PM
> > Subject: Re: [OMPI devel] Performance analysis proposal
> >
> > Hi, Christoph
> >
> > Please, check https://github.com/open-mpi/ompi/wiki/Request-refactoring-
> test for the testing methodology and
> > https://github.com/open-mpi/2016-summer-perf-testing
> > for examples and launch scripts.
> >
> > 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de > :
> >
> >
> > Hello,
> >
> > I just came over this and would like to contribute some results from our
> system(s).
> > Are there any specific configure options you want to see enabled beside
> --enable-mpi-thread-multiple?
> > How to commit results?
> >
> > Best
> > Christoph Niethammer
> >
> >
> >
> > - Original Message -
> > From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com >
> > To: "Open MPI Developers" < devel@lists.open-mpi.org >
> > Sent: Friday, July 29, 2016 8:41:06 PM
> > Subject: Re: [OMPI de

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

Sufficient. Probably I missed it. No need to do anything.

2016-08-26 21:31 GMT+07:00 Jeff Squyres (jsquyres) :

> Just curious: is https://github.com/open-mpi/2016-summer-perf-testing not
> sufficient?
>
>
>
> > On Aug 26, 2016, at 10:28 AM, Artem Polyakov  wrote:
> >
> > I'd prefer it to be created.
> >
> > 2016-08-26 20:59 GMT+07:00 Jeff Squyres (jsquyres) :
> > Sorry for jumping in so late.
> >
> > Honestly, there's no problem with making a repo in the open-mpi github
> org.  It's just as trivial to make one there as anywhere else.
> >
> > Let me know if you want one.
> >
> >
> > > On Aug 26, 2016, at 8:46 AM, Artem Polyakov 
> wrote:
> > >
> > > I've marked the first week.
> > >
> > > 2016-08-26 19:26 GMT+07:00 George Bosilca :
> > > Let's go regular for a period and then adapt.
> > >
> > > For everybody interested in the performance discussion, I setup a
> doodle for next week. The dates themselves are not important, we need a
> regular timeslot. Please answer with the idea that we do 4 weeks in a row
> and then assess the situation and dece if we need to continue, decrease the
> frequency or declare the problem solved. Here is the participation link:
> http://doodle.com/poll/w4fkb9gr3h2q5p6v
> > >
> > >   George.
> > >
> > >
> > > On Fri, Aug 26, 2016 at 6:39 AM, Artem Polyakov 
> wrote:
> > > This is a good idea. We also have data with SM/Vader to discuss. I'll
> send them later this week.
> > >
> > > Do you think of regular calls or per agreement?
> > >
> > > пятница, 26 августа 2016 г. пользователь George Bosilca написал:
> > > We are serious about this. However, we not only have to define a set
> of meaningful tests (which we don't have yet) but also decide the
> conditions in which they are executed, and more critically what additional
> information we need to make them reproducible, understandable and
> comparable.
> > >
> > > We started discussion on these topics during the developers meeting
> few weeks ago, but we barely define what we think will be necessary for
> trivial tests such as single threaded bandwidth. It might be worth having a
> regular phone call (in addition to the Tuesday morning) to make progress.
> > >
> > >   George.
> > >
> > >
> > > On Thu, Aug 25, 2016 at 9:37 PM, Artem Polyakov 
> wrote:
> > > If we are serious about this problem I don't see why we can't create a
> repo for this data and keep the history of all measurements.
> > >
> > > Is there any chance that we will not came up with well defined set of
> tests and drop the ball here?
> > >
> > > пятница, 26 августа 2016 г. пользователь George Bosilca написал:
> > >
> > > Arm repo is a good location until we converge to a well-defined set of
> tests.
> > >
> > >   George.
> > >
> > >
> > > On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov 
> wrote:
> > > That's a good question. I have results myself and I don't know where
> to place them.
> > > I think that Arm's repo is not a right place to collect the data.
> > >
> > > Jeff, can we create the repo in open mpi organization on github or do
> we have something appropriate already?
> > >
> > > четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал:
> > >
> > > Hi Artem,
> > >
> > > Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader
> performance regression patch under
> > > https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc9c0847b3
> 091601919f.patch
> > > and master. I will do the 2.0.1rc in the next days as well.
> > >
> > > Is it possible to add me to the results repository at github or should
> I fork and request you to pull?
> > >
> > > Best
> > > Christoph
> > >
> > >
> > > - Original Message -
> > > From: "Artem Polyakov" 
> > > To: "Open MPI Developers" 
> > > Sent: Tuesday, August 23, 2016 5:13:30 PM
> > > Subject: Re: [OMPI devel] Performance analysis proposal
> > >
> > > Hi, Christoph
> > >
> > > Please, check https://github.com/open-mpi/
> ompi/wiki/Request-refactoring-test for the testing methodology and
> > > https://github.com/open-mpi/2016-summer-perf-testing
> > > for examples and launch scripts.
> > >
> > > 2016-08-23 21:20 GMT+07:00 Christo

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

Howard,

can you link to commits you are referring?
Do you mean this one for example:
https://github.com/open-mpi/ompi/commit/15098161a331168c66b29a696522fe52c8b2d8f5
?

2016-12-01 15:28 GMT-08:00 Howard Pritchard :

> Hi Gilles
>
> I didn't see a merge commit for all these commits,
> hence my concern that it was a mistake.
>
> In general it's better to pull in commits via PR process.
>
> Howard
>
>
> Am Donnerstag, 1. Dezember 2016 schrieb Gilles Gouaillardet :
>
>> fwiw, the major change is in https://github.com/open-mpi
>> /ompi/commit/c9aeccb84e4626c350af4daa974d37775db5b25e
>> and i did spend quite some time testing it.
>>
>> if you are facing some unexpected issues with the latest master, you
>> might try to revert this locally and see if it helps
>>
>> Cheers,
>>
>> Gilles
>>
>> On Friday, December 2, 2016, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>>
>>> Howard,
>>>
>>> i pushed a bunch of commits yesterday, and that was not an accident.
>>> you might be referring https://github.com/o
>>> pen-mpi/ompi/commit/cb55c88a8b7817d5891ff06a447ea190b0e77479 but it has
>>> already been reverted 9 days ago with https://github.com/open-m
>>> pi/ompi/commit/1e2019ce2a903be24361b3424d8e98d27e941c6c
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On Friday, December 2, 2016, r...@open-mpi.org  wrote:
>>>
 I should add, FWIW: I’m working with the HEAD of master right now, and
 not seeing any problems.

 On Dec 1, 2016, at 2:10 PM, r...@open-mpi.org wrote:

 ?? I see a bunch of commits that were all collected in a single PR from
 Gilles yesterday - is that what you are referring to?

 On Dec 1, 2016, at 1:58 PM, Howard Pritchard 
 wrote:

 Hi Folks,

 Just an FYI it looks like a bunch of commits may have been accidentally
 pushed to
 master sometime in the past day.  You may not want to merge
 origin/master (if origin is
 how you reference https://github.com/open-mpi/omp) into your master or
 rebase off
 of it until we get some clarity on what has happened.

 Howard

 ___
 devel mailing list
 devel@lists.open-mpi.org
 https://rfd.newmexicoconsortium.org/mailman/listinfo/devel




> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

All systems are different and it is hard to compete in coverage with our
set of Jenkins' :).

2016-12-01 14:51 GMT-08:00 r...@open-mpi.org :

> FWIW: I verified it myself, and it was fine on my systems
>
> On Dec 1, 2016, at 2:46 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
> fwiw, the major change is in https://github.com/open-mpi/ompi/commit/
> c9aeccb84e4626c350af4daa974d37775db5b25e
> and i did spend quite some time testing it.
>
> if you are facing some unexpected issues with the latest master, you might
> try to revert this locally and see if it helps
>
> Cheers,
>
> Gilles
>
> On Friday, December 2, 2016, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Howard,
>>
>> i pushed a bunch of commits yesterday, and that was not an accident.
>> you might be referring https://github.com/open-mpi/ompi/commit/cb55c88a8b
>> 7817d5891ff06a447ea190b0e77479 but it has already been reverted 9 days
>> ago with https://github.com/open-mpi/ompi/commit/1e2019ce2a903be
>> 24361b3424d8e98d27e941c6c
>>
>> Cheers,
>>
>> Gilles
>>
>> On Friday, December 2, 2016, r...@open-mpi.org  wrote:
>>
>>> I should add, FWIW: I’m working with the HEAD of master right now, and
>>> not seeing any problems.
>>>
>>> On Dec 1, 2016, at 2:10 PM, r...@open-mpi.org wrote:
>>>
>>> ?? I see a bunch of commits that were all collected in a single PR from
>>> Gilles yesterday - is that what you are referring to?
>>>
>>> On Dec 1, 2016, at 1:58 PM, Howard Pritchard 
>>> wrote:
>>>
>>> Hi Folks,
>>>
>>> Just an FYI it looks like a bunch of commits may have been accidentally
>>> pushed to
>>> master sometime in the past day.  You may not want to merge
>>> origin/master (if origin is
>>> how you reference https://github.com/open-mpi/omp) into your master or
>>> rebase off
>>> of it until we get some clarity on what has happened.
>>>
>>> Howard
>>>
>>> ___
>>> devel mailing list
>>> devel@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>>
>>>
>>>
>>> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
>
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

But I guess that we can verify that things are not broken using other PR's.
Looks that all is good: https://github.com/open-mpi/ompi/pull/2493

2016-12-01 15:38 GMT-08:00 Artem Polyakov :

> All systems are different and it is hard to compete in coverage with our
> set of Jenkins' :).
>
> 2016-12-01 14:51 GMT-08:00 r...@open-mpi.org :
>
>> FWIW: I verified it myself, and it was fine on my systems
>>
>> On Dec 1, 2016, at 2:46 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>>
>> fwiw, the major change is in https://github.com/open-mpi
>> /ompi/commit/c9aeccb84e4626c350af4daa974d37775db5b25e
>> and i did spend quite some time testing it.
>>
>> if you are facing some unexpected issues with the latest master, you
>> might try to revert this locally and see if it helps
>>
>> Cheers,
>>
>> Gilles
>>
>> On Friday, December 2, 2016, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com> wrote:
>>
>>> Howard,
>>>
>>> i pushed a bunch of commits yesterday, and that was not an accident.
>>> you might be referring https://github.com/o
>>> pen-mpi/ompi/commit/cb55c88a8b7817d5891ff06a447ea190b0e77479 but it has
>>> already been reverted 9 days ago with https://github.com/open-m
>>> pi/ompi/commit/1e2019ce2a903be24361b3424d8e98d27e941c6c
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>> On Friday, December 2, 2016, r...@open-mpi.org  wrote:
>>>
>>>> I should add, FWIW: I’m working with the HEAD of master right now, and
>>>> not seeing any problems.
>>>>
>>>> On Dec 1, 2016, at 2:10 PM, r...@open-mpi.org wrote:
>>>>
>>>> ?? I see a bunch of commits that were all collected in a single PR from
>>>> Gilles yesterday - is that what you are referring to?
>>>>
>>>> On Dec 1, 2016, at 1:58 PM, Howard Pritchard 
>>>> wrote:
>>>>
>>>> Hi Folks,
>>>>
>>>> Just an FYI it looks like a bunch of commits may have been accidentally
>>>> pushed to
>>>> master sometime in the past day.  You may not want to merge
>>>> origin/master (if origin is
>>>> how you reference https://github.com/open-mpi/omp) into your master or
>>>> rebase off
>>>> of it until we get some clarity on what has happened.
>>>>
>>>> Howard
>>>>
>>>> ___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>>>
>>>>
>>>>
>>>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>>
>>
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

Exactly this commit
https://github.com/open-mpi/ompi/commit/15098161a331168c66b29a696522fe52c8b2d8f5
wan't part of that PR

2016-12-01 16:21 GMT-08:00 Howard Pritchard :

> Ralph,
>
> I don't know how it happened but if you do
>
> git log --oneline --topo-order
>
> you don't see a Merge pull request #2488 in the history for master.
>
> Howard
>
>
> 2016-12-01 16:59 GMT-07:00 r...@open-mpi.org :
>
>> Ummm...guys, it was done via PR. I saw it go by, and it was all done to
>> procedure:
>>
>> https://github.com/open-mpi/ompi/pull/2488
>>
>> So please don’t jump to conclusions
>>
>> On Dec 1, 2016, at 3:49 PM, Artem Polyakov  wrote:
>>
>> But I guess that we can verify that things are not broken using other
>> PR's.
>> Looks that all is good: https://github.com/open-mpi/ompi/pull/2493
>>
>> 2016-12-01 15:38 GMT-08:00 Artem Polyakov :
>>
>>> All systems are different and it is hard to compete in coverage with our
>>> set of Jenkins' :).
>>>
>>> 2016-12-01 14:51 GMT-08:00 r...@open-mpi.org :
>>>
>>>> FWIW: I verified it myself, and it was fine on my systems
>>>>
>>>> On Dec 1, 2016, at 2:46 PM, Gilles Gouaillardet <
>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>
>>>> fwiw, the major change is in https://github.com/open-mpi
>>>> /ompi/commit/c9aeccb84e4626c350af4daa974d37775db5b25e
>>>> and i did spend quite some time testing it.
>>>>
>>>> if you are facing some unexpected issues with the latest master, you
>>>> might try to revert this locally and see if it helps
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> On Friday, December 2, 2016, Gilles Gouaillardet <
>>>> gilles.gouaillar...@gmail.com> wrote:
>>>>
>>>>> Howard,
>>>>>
>>>>> i pushed a bunch of commits yesterday, and that was not an accident.
>>>>> you might be referring https://github.com/o
>>>>> pen-mpi/ompi/commit/cb55c88a8b7817d5891ff06a447ea190b0e77479 but it
>>>>> has already been reverted 9 days ago with https://github.com/open-m
>>>>> pi/ompi/commit/1e2019ce2a903be24361b3424d8e98d27e941c6c
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Gilles
>>>>>
>>>>> On Friday, December 2, 2016, r...@open-mpi.org 
>>>>> wrote:
>>>>>
>>>>>> I should add, FWIW: I’m working with the HEAD of master right now,
>>>>>> and not seeing any problems.
>>>>>>
>>>>>> On Dec 1, 2016, at 2:10 PM, r...@open-mpi.org wrote:
>>>>>>
>>>>>> ?? I see a bunch of commits that were all collected in a single PR
>>>>>> from Gilles yesterday - is that what you are referring to?
>>>>>>
>>>>>> On Dec 1, 2016, at 1:58 PM, Howard Pritchard 
>>>>>> wrote:
>>>>>>
>>>>>> Hi Folks,
>>>>>>
>>>>>> Just an FYI it looks like a bunch of commits may have been
>>>>>> accidentally pushed to
>>>>>> master sometime in the past day.  You may not want to merge
>>>>>> origin/master (if origin is
>>>>>> how you reference https://github.com/open-mpi/omp) into your master
>>>>>> or rebase off
>>>>>> of it until we get some clarity on what has happened.
>>>>>>
>>>>>> Howard
>>>>>>
>>>>>> ___
>>>>>> devel mailing list
>>>>>> devel@lists.open-mpi.org
>>>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>>>>>
>>>>>>
>>>>>>
>>>>>> ___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>>>
>>>>
>>>>
>>>> ___
>>>> devel mailing list
>>>> devel@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>>>
>>>
>>>
>>>
>>> --
>>> С Уважением, Поляков Артем Юрьевич
>>> Best regards, Artem Y. Polyakov
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>>
>>
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

+1 to Paul.

I had to go git-bisect OMPI only several times but it always was a
non-trivial task. PR's are grouping commit's logically and are good for the
bookkeeping.
Also you never know what will a "trivial fix" turn into and in what
circumstances/configurations.
IMO all changes needs to go through CI, no exceptions.

2016-12-01 16:41 GMT-08:00 Paul Hargrove :

>
> On Thu, Dec 1, 2016 at 4:25 PM, Gilles Gouaillardet 
> wrote:
> [...]
>
>> git checkout master
>>
>> git merge --ff-only topic/misc_fixes
>>
>> git push origin master
>>
> [...]
>
>
> Gilles,
>
> You characterized the merge commit has having "close to zero added value"
> to you - but in this instance it would have saved you and others a
> non-trivial amount of time in email.
>
> Additionally, in projects I work on we value that merge commit as a "cut
> line" if we ever need to revert an entire PR for some reason.  Using
> git-bisect such that one includes or excludes the entire PR is also a
> justification for keeping the merge commit.  So my opinion is that you
> should have omitted "--ff-only" and entered a commit message that at least
> identified the PR number.
>
>
>> though this does not generate a git commit, github.com is smart enough
>> to figure this out and marks the PR as merged.
>>
>
> FWIW: "smart enough" is simply a detection that the last commit in the PR
> has become an ancestor of the current HEAD.
>
> -Paul
>
> --
> Paul H. Hargrove  phhargr...@lbl.gov
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> <(510)%20495-2352>
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> <(510)%20486-6900>
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] heads up about OMPI/master

[Artem Polyakov]

With regard to timezone - we have developers in close timezones, so I don't
think this is a reasonable argument.

2016-12-01 16:49 GMT-08:00 Artem Polyakov :

> +1 to Paul.
>
> I had to go git-bisect OMPI only several times but it always was a
> non-trivial task. PR's are grouping commit's logically and are good for the
> bookkeeping.
> Also you never know what will a "trivial fix" turn into and in what
> circumstances/configurations.
> IMO all changes needs to go through CI, no exceptions.
>
> 2016-12-01 16:41 GMT-08:00 Paul Hargrove :
>
>>
>> On Thu, Dec 1, 2016 at 4:25 PM, Gilles Gouaillardet 
>> wrote:
>> [...]
>>
>>> git checkout master
>>>
>>> git merge --ff-only topic/misc_fixes
>>>
>>> git push origin master
>>>
>> [...]
>>
>>
>> Gilles,
>>
>> You characterized the merge commit has having "close to zero added value"
>> to you - but in this instance it would have saved you and others a
>> non-trivial amount of time in email.
>>
>> Additionally, in projects I work on we value that merge commit as a "cut
>> line" if we ever need to revert an entire PR for some reason.  Using
>> git-bisect such that one includes or excludes the entire PR is also a
>> justification for keeping the merge commit.  So my opinion is that you
>> should have omitted "--ff-only" and entered a commit message that at least
>> identified the PR number.
>>
>>
>>> though this does not generate a git commit, github.com is smart enough
>>> to figure this out and marks the PR as merged.
>>>
>>
>> FWIW: "smart enough" is simply a detection that the last commit in the PR
>> has become an ancestor of the current HEAD.
>>
>> -Paul
>>
>> --
>> Paul H. Hargrove  phhargr...@lbl.gov
>> Computer Languages & Systems Software (CLaSS) Group
>> Computer Science Department   Tel: +1-510-495-2352
>> <(510)%20495-2352>
>> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>> <(510)%20486-6900>
>>
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Mellanox Jenkins

[Artem Polyakov]

Brian, I'm going to push for the fix tonight. If won't work - we will do as
you advised.

2017-06-21 17:23 GMT-07:00 Barrett, Brian via devel <
devel@lists.open-mpi.org>:

> In the mean time, is it possible to disable the jobs that listen for pull
> requests on Open MPI’s repos?  I’m trying to get people out of the habit of
> ignoring CI results, so no results are better than failed results :/.
>
> Brian
>
> > On Jun 21, 2017, at 1:49 PM, Jeff Squyres (jsquyres) 
> wrote:
> >
> > Thanks Josh.
> >
> >> On Jun 21, 2017, at 2:18 PM, Joshua Ladd  wrote:
> >>
> >> OMPI Developers,
> >>
> >> We are aware of the issue currently affecting the Mellanox Jenkins
> servers. The issue is being addressed and we hope it will be resolved soon.
> We apologize for the inconvenience and thank you for your patience.
> >>
> >> Best,
> >>
> >> Josh Ladd
> >>
> >>
> >> ___
> >> devel mailing list
> >> devel@lists.open-mpi.org
> >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> >
> >
> > --
> > Jeff Squyres
> > jsquy...@cisco.com
> >
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] [3.0.0rc1] PMIX ERROR: UNPACK-INADEQUATE-SPACE

[Artem Polyakov]

Hello, Paul.
How OMPI was configured ? Were you by chance using external PMIx?

пн, 3 июля 2017 г. в 18:43, Paul Hargrove :

>
> On (at least) two different hosts (both Linux, one x86-64 and one ppc64el)
> I am seeing a failure to launch ring_c with errors like those shown below.
>
> -Paul
>
> $ mpirun -mca btl sm,self -np 2 examples/ring_c
> [pcp-d-1:02255] PMIX ERROR: UNPACK-INADEQUATE-SPACE in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/dstore/pmix_esh.c
> at line 2477
> [pcp-d-1:02255] PMIX ERROR: ERROR in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/dstore/pmix_esh.c
> at line 2024
> [pcp-d-1:02255] PMIX ERROR: UNPACK-INADEQUATE-SPACE in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/dstore/pmix_esh.c
> at line 1150
> [pcp-d-1:02255] PMIX ERROR: UNPACK-INADEQUATE-SPACE in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/common/pmix_jobdata.c
> at line 112
> [pcp-d-1:02255] PMIX ERROR: UNPACK-INADEQUATE-SPACE in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/common/pmix_jobdata.c
> at line 392
> [pcp-d-1:02255] PMIX ERROR: UNPACK-INADEQUATE-SPACE in file
> /home/phargrov/OMPI/openmpi-3.0.0rc1-linux-x86_64-pathcc-6/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/server/pmix_server.c
> at line 518
>
> --
> Paul H. Hargrove  phhargr...@lbl.gov
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

-- 
- Best regards, Artem Polyakov (Mobile mail)
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] v3.0.x / v3.x branch mixup

[Artem Polyakov]

Brian,
We will address.

пт, 7 июля 2017 г. в 14:25, Barrett, Brian via devel <
devel@lists.open-mpi.org>:

> Hi all -
>
> Earlier this week, we discovered that a couple of pull requests had been
> posted against the deprecated v3.x branch (instead of the active v3.0.x
> branch).  Worse, Github allowed me to merge those requests, despite Github
> reporting that nobody had permissions to write to the branch (oddly, Howard
> can not push to the branch).  After some auditing, it appears that there is
> only one commit in the v3.x branch that is not in the v3.0.x branch:
>
>
> https://github.com/open-mpi/ompi/commit/8a4487900831f9b0dbed1f00cb9cc30921988bc2
>
> Boris, can you file a PR against v3.0.x if this patch is still desired for
> the v3.0.0 release?
>
> Apologies to all; we’ll do better for the fall release.
>
> Brian
>
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

-- 
- Best regards, Artem Polyakov (Mobile mail)
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Yoda SPML and master/v3.0.0

[Artem Polyakov]

We will try to address till the end of the week.

чт, 13 июля 2017 г. в 18:37, Barrett, Brian via devel <
devel@lists.open-mpi.org>:

> Mellanox developers -
>
> The btl sm header leak in the yoda spml brought up questions about the
> status of the yoda spml.  My understanding was that Mellanox was going to
> remove it after the decision that we didn’t require supporting btl
> transports and Mellanox no longer wanting to support yoda.  But it looks
> like yoda is still in master and v3.0.x.  Can we remove it?  Is it possible
> to get a patch in the next couple of days from Mellanox?
>
> Thanks,
>
> Brian
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

-- 
- Best regards, Artem Polyakov (Mobile mail)
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Issue/PR tagging

[Artem Polyakov]

Brian,

Have you had a chance to put this on the wiki? If so - can you send the
link - I can't find it.

2017-07-19 16:47 GMT-07:00 Barrett, Brian via devel <
devel@lists.open-mpi.org>:

> I’ll update the wiki (and figure out where on our wiki to put more general
> information), but the basics are:
>
> If you find a bug, file an issue.  Add Target:v??? labels for any branch
> it impacts.  If we decide later not to fix the issue on a branch, we’ll
> remove the label
> Open/find an issue for any PR going to release branches.  That issue can
> (possibly should, if the issue impacts multiple branches) have multiple
> Target:v??? labels
> If a PR is for a release branch (ie, it’s immediate target to merge to is
> a release branch), please add a Target:v??? label and reference the issue
> If a PR is for master, it can reference an issue (if there’s an issue
> associated with it), but should not have a Target:v??? label
> If an issue is fixed in master, but not merged into branches, don’t close
> the issue
>
> I think that’s about it.  There’s some workflows we want to build to
> automate enforcing many of these things, but for now, it’s just hints to
> help the RMs not lose track of issues.
>
> Brian
>
> > On Jul 19, 2017, at 12:18 PM, r...@open-mpi.org wrote:
> >
> > Hey folks
> >
> > I know we made some decisions last week about how to tag issues and PRs
> to make things easier to track for release branches, but the wiki notes
> don’t cover what we actually decided to do. Can someone briefly summarize?
> I honestly have forgotten if we tag issues, or tag PRs
> >
> > Ralph
> >
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

Jeff and others,

1. The benchmark was updated to support shared memory case.
2. The wiki was updated with the benchmark description:
https://github.com/open-mpi/ompi/wiki/Request-refactoring-test#benchmark-prototype

Let me know if we want to put this prototype to some general place. I think
it makes sense.


2016-07-28 17:22 GMT+07:00 Jeff Squyres (jsquyres) :

> On Jul 28, 2016, at 2:52 AM, Sreenidhi Bharathkar Ramesh via devel <
> devel@lists.open-mpi.org> wrote:
> >
> >> For Open MPI, it's basically THREAD_MULTIPLE and not-THREAD_MULTIPLE.
> I.e., there's no real difference between SINGLE, SERIALIZED, FUNNELED
> >
> > We were assuming that there would be cost due to
> > locking/synchronization in FUNNELED and SERIALIZED also.
>
> The MPI spec says that *applications* must ensure the correctness for
> FUNNELED and SERIALIZED -- so Open MPI does not provide additional locking
> for these levels.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

Thank you, Arm!

Good to have vader results (I haven't tried it myself yet). Few
comments/questions:
1. I guess we also want to have 1-threaded performance for the "baseline"
reference.
2. Have you tried to run with openib, as I mentioned on the call I had some
problems with it and I'm curious if you can reproduce in your environment.

Github issue sounds good for me!

2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) <
apati...@cisco.com>:

> I added some result to
> https://github.com/open-mpi/2016-summer-perf-testing
>
> The result shows much better performance from 2.0.0 and master over 1.10.3
> for vader. The test ran with Artem’s version of benchmark on OB1, single
> node, bind to socket.
>
> We should have a place to discuss/comment/collaborate on results. Should I
> open an issue on that repo? So we can have github style
> commenting/referencing.
>
>
> Arm
>
>
>
>
> On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" <
> devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com> wrote:
>
> >On Jul 28, 2016, at 6:28 AM, Artem Polyakov  wrote:
> >>
> >> Jeff and others,
> >>
> >> 1. The benchmark was updated to support shared memory case.
> >> 2. The wiki was updated with the benchmark description:
> https://github.com/open-mpi/ompi/wiki/Request-refactoring-test#benchmark-prototype
> >
> >Sweet -- thanks!
> >
> >> Let me know if we want to put this prototype to some general place. I
> think it makes sense.
> >
> >I just created:
> >
> >https://github.com/open-mpi/2016-summer-perf-testing
> >
> >Want to put it there?
> >
> >Arm just ran a bunch of tests today and will be committing a bunch of
> results in there shortly.
> >
> >--
> >Jeff Squyres
> >jsquy...@cisco.com
> >For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >
> >___
> >devel mailing list
> >devel@lists.open-mpi.org
> >https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] Performance analysis proposal

[Artem Polyakov]

P.S. For the future reference we also need to keep launch scripts that were
used to be able to carefully reproduce. Jeff mentioned that on the wiki
page IFRC.

2016-07-29 12:42 GMT+07:00 Artem Polyakov :

> Thank you, Arm!
>
> Good to have vader results (I haven't tried it myself yet). Few
> comments/questions:
> 1. I guess we also want to have 1-threaded performance for the "baseline"
> reference.
> 2. Have you tried to run with openib, as I mentioned on the call I had
> some problems with it and I'm curious if you can reproduce in your
> environment.
>
> Github issue sounds good for me!
>
> 2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) <
> apati...@cisco.com>:
>
>> I added some result to
>> https://github.com/open-mpi/2016-summer-perf-testing
>>
>> The result shows much better performance from 2.0.0 and master over
>> 1.10.3 for vader. The test ran with Artem’s version of benchmark on OB1,
>> single node, bind to socket.
>>
>> We should have a place to discuss/comment/collaborate on results. Should
>> I open an issue on that repo? So we can have github style
>> commenting/referencing.
>>
>>
>> Arm
>>
>>
>>
>>
>> On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" <
>> devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com> wrote:
>>
>> >On Jul 28, 2016, at 6:28 AM, Artem Polyakov  wrote:
>> >>
>> >> Jeff and others,
>> >>
>> >> 1. The benchmark was updated to support shared memory case.
>> >> 2. The wiki was updated with the benchmark description:
>> https://github.com/open-mpi/ompi/wiki/Request-refactoring-test#benchmark-prototype
>> >
>> >Sweet -- thanks!
>> >
>> >> Let me know if we want to put this prototype to some general place. I
>> think it makes sense.
>> >
>> >I just created:
>> >
>> >https://github.com/open-mpi/2016-summer-perf-testing
>> >
>> >Want to put it there?
>> >
>> >Arm just ran a bunch of tests today and will be committing a bunch of
>> results in there shortly.
>> >
>> >--
>> >Jeff Squyres
>> >jsquy...@cisco.com
>> >For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> >
>> >___
>> >devel mailing list
>> >devel@lists.open-mpi.org
>> >https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>> ___
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] OPAL timing framework

[Artem Polyakov]

Hello,

I would like to introduce OMPI timing framework that was included into the
trunk yesterday (r32738). The code is new so if you'll hit some bugs - just
let me know.

The framework consists of the set of macro's and routines for internal OMPI
usage + standalone tool mpisync and few additional scripts: mpirun_prof and
ompi_timing_post. The set of features is very basic and I am open for
discussion of new things that are desirable there.

To enable framework compilation you should configure OMPI with
--enable-timing option. If the option was passed to ./configure, standalone
tools and scripts will be installed into /bin.

The timing code is located in OPAL (opal/utils/timing.[ch]). There is a set
of macro's that should be used to preprocess out all mentions of the timing
code in case it wasn't requested with --enable-timing:
OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t".
OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler
"t".
OPAL_TIMING_INIT(t) - initialize timing handler "t"
OPAL_TIMING_EVENT(x) - printf-like event declaration similar to OPAL_OUTPUT.
The information about the event will be quickly inserted into the linked
list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX.
The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and
overhead (time to malloc and prepare the bucket) is accounted in
corresponding list element. It might be excluded from the timing results
(controlled by OMPI_MCA_opal_timing_overhead parameter).
OPAL_TIMING_REPORT(enable, t, prefix) - prepare and print out timing
information. If OMPI_MCA_opal_timing_file was specified the output will go
to that file. In other case the output will be directed using opal_output,
each line will be prefixed with "prefix" to ease grep'ing. "enable" is a
boolean/integer variable that is used for runtime selection of what should
be reported.
OPAL_TIMING_RELEASE(t) - the counterpart for OPAL_TIMING_INIT.

There are several examples in OMPI code. And here is another simple example:
OPAL_TIMING_DECLARE(tm);
OPAL_TIMING_INIT(&tm);
...
OPAL_TIMING_EVENT((&tm,"Begin of timing: %s",
ORTE_NAME_PRINT(&(peer->name)) ));

OPAL_TIMING_EVENT((&tm,"Next timing event with condition x = %d", x ));
...
OPAL_TIMING_EVENT((&tm,"Finish"));
OPAL_TIMING_REPORT(enable_var, &tm,"MPI Init");
OPAL_TIMING_RELEASE(&tm);


An output from all OMPI processes (mpirun, orted's, user processes) is
merged together. NTP provides 1 millisecond - 100 microsecond level of
precision. This may not be sufficient to order events globally.
To help developers extract the most realistic picture of what is going on,
additional time synchronisation might be performed before profiling. The
mpisync program should be runned 1-user-process-per-node to acquire the
file with time offsets relative to HNP of each node. If the cluster runs
over Gig Ethernet the precision will be 30-50 microseconds, in case of
Infiniband - 4 microseconds. mpisync produces output file that might be
readed and used by timing framework (OMPI_MCA_opal_clksync_file parameter).
The bad news is that this synchronisation is not enough because of
different clock skew on different nodes. Additional periodical
synchronisation is needed. This is planned for the near future (me and
Ralph discussing possible ways now).

the mpirun_prof & ompi_timing_post script may be used to automate clock
synchronisation in following manner:
export OMPI_MCA_ompi_timing=true
export OMPI_MCA_orte_oob_timing=true
export OMPI_MCA_orte_rml_timing=true
export OMPI_MCA_opal_timing_file=timing.out
mpirun_prof  ./mpiprog
ompi_timing_post timing.out

ompi_timing_post will simply sort the events and made all times to be
relative to the first one.

-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] OPAL timing framework

[Artem Polyakov]

Jeff, thank you for the feedback! All of mentioned issues are clear and I
will fix them shortly.

One important thing that needs additional discussion is compile-time vs
runtime selection. Ralph, what do you think about that? Several of issues
depends on that decision.

2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres) :

> I have a few comments:
>
> - This looks nice.  Thanks for the contribution.
>
> - I notice that the ORTE timing stuff is now a compile-time decision, not
> a run-time decision.  Do we care that we've now taken away the ability for
> users to do timings in a production build?

- "clksync" -- can we use "clocksync"?  It's only 2 letters.  We tend to
> use real words in the OMPI code base; unnecessary abbreviation should be
> avoided.


> - r32738 introduced a several files into the code base that have no
> copyrights, and do not have the standard OMPI copyright header block.
> Please fix.
>
> - There's no documentation on how to use mpisync, mpirun_prof, or
> ompi_timing_post, even though they're installed when you --enable-timing.
> What are these 3 executables?  Can we get man pages?
>
I post their description in the first e-mail. Sure I can prepare man pages
for them,


>
> - What's the purpose of the MCA param orte_rml_base_timing?  A *quick*
> look through the code seems to indicate that it is ignored.
>
> - What's the purpose of the MCA params opal_clksync_file,
> opal_timing_file, and opal_timing_overhead?  E.g., what is a "clksync"
> file, what is it for, and what is its format?  Does the user have to
> provide one?  If so, how to you get one?  Or is it an output file?
> ...etc.  The brief descriptions given in the MCA help strings don't really
> provide enough information for someone who has no idea what the timing
> stuff is.  Also, can those 3 params have a common prefix?  I.e., it's not
> obvious that opal_clksync_file is related to opal_timing_* at all.


> - A *quick* look at ompi/tools/mpisync shows that a bunch of that code
> came from an external project.  Is the license compatible with OMPI's
> license?  What do we need to do to conform to their license?
>
> - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it be
> OPAL_UTIL_TIMINGS_H?
>
> - There's commented-out code in opal/util/timings.h.
>
> - There's no doxygen-style documentation in opal/util/timings.h to tell
> developers how to use it.
>
> - There's "TODO" comments in opal/util/timings.c; should those be fixed?
>
> - opal_config.h should be the first include in opal/util/timings.c.
>
> - If timing support is not to be compiled in, then opal/util/timings.c
> should not be be compiled via the Makefile.am (rather than entirely #if'ed
> out).
>
> It looks like this work is about 95% complete.  Finishing the remaining 5%
> would make it great and genuinely useful to the rest of the code base.
>
> Thanks!
>
>
>
> On Sep 16, 2014, at 10:20 AM, Artem Polyakov  wrote:
>
> > Hello,
> >
> > I would like to introduce OMPI timing framework that was included into
> the trunk yesterday (r32738). The code is new so if you'll hit some bugs -
> just let me know.
> >
> > The framework consists of the set of macro's and routines for internal
> OMPI usage + standalone tool mpisync and few additional scripts:
> mpirun_prof and ompi_timing_post. The set of features is very basic and I
> am open for discussion of new things that are desirable there.
> >
> > To enable framework compilation you should configure OMPI with
> --enable-timing option. If the option was passed to ./configure, standalone
> tools and scripts will be installed into /bin.
> >
> > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a
> set of macro's that should be used to preprocess out all mentions of the
> timing code in case it wasn't requested with --enable-timing:
> > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t".
> > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler
> "t".
> > OPAL_TIMING_INIT(t) - initialize timing handler "t"
> > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to
> OPAL_OUTPUT.
> > The information about the event will be quickly inserted into the linked
> list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX.
> > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and
> overhead (time to malloc and prepare the bucket) is accounted in
> corresponding list element. It might be excluded from the timing results
> (controlled by OMPI_MCA_opal_t

Re: [OMPI devel] OPAL timing framework

[Artem Polyakov]

Got that. Thank you!

четверг, 18 сентября 2014 г. пользователь Ralph Castain написал:

> I believe compile-time is preferable as there is a non-zero time impact of
> enabling this code. It's really more for developers to improve scalability
> - if a user is actually interested, I think it isn't that hard for them to
> configure it.
>
>
> On Sep 18, 2014, at 7:16 AM, Artem Polyakov  > wrote:
>
> Jeff, thank you for the feedback! All of mentioned issues are clear and I
> will fix them shortly.
>
> One important thing that needs additional discussion is compile-time vs
> runtime selection. Ralph, what do you think about that? Several of issues
> depends on that decision.
>
> 2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres)  >:
>
>> I have a few comments:
>>
>> - This looks nice.  Thanks for the contribution.
>>
>> - I notice that the ORTE timing stuff is now a compile-time decision, not
>> a run-time decision.  Do we care that we've now taken away the ability for
>> users to do timings in a production build?
>
> - "clksync" -- can we use "clocksync"?  It's only 2 letters.  We tend to
>> use real words in the OMPI code base; unnecessary abbreviation should be
>> avoided.
>
>
>> - r32738 introduced a several files into the code base that have no
>> copyrights, and do not have the standard OMPI copyright header block.
>> Please fix.
>>
>> - There's no documentation on how to use mpisync, mpirun_prof, or
>> ompi_timing_post, even though they're installed when you --enable-timing.
>> What are these 3 executables?  Can we get man pages?
>>
> I post their description in the first e-mail. Sure I can prepare man pages
> for them,
>
>
>>
>> - What's the purpose of the MCA param orte_rml_base_timing?  A *quick*
>> look through the code seems to indicate that it is ignored.
>>
>> - What's the purpose of the MCA params opal_clksync_file,
>> opal_timing_file, and opal_timing_overhead?  E.g., what is a "clksync"
>> file, what is it for, and what is its format?  Does the user have to
>> provide one?  If so, how to you get one?  Or is it an output file?
>> ...etc.  The brief descriptions given in the MCA help strings don't really
>> provide enough information for someone who has no idea what the timing
>> stuff is.  Also, can those 3 params have a common prefix?  I.e., it's not
>> obvious that opal_clksync_file is related to opal_timing_* at all.
>
>
>> - A *quick* look at ompi/tools/mpisync shows that a bunch of that code
>> came from an external project.  Is the license compatible with OMPI's
>> license?  What do we need to do to conform to their license?
>>
>> - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it
>> be OPAL_UTIL_TIMINGS_H?
>>
>> - There's commented-out code in opal/util/timings.h.
>>
>> - There's no doxygen-style documentation in opal/util/timings.h to tell
>> developers how to use it.
>>
>> - There's "TODO" comments in opal/util/timings.c; should those be fixed?
>>
>> - opal_config.h should be the first include in opal/util/timings.c.
>>
>> - If timing support is not to be compiled in, then opal/util/timings.c
>> should not be be compiled via the Makefile.am (rather than entirely #if'ed
>> out).
>>
>> It looks like this work is about 95% complete.  Finishing the remaining
>> 5% would make it great and genuinely useful to the rest of the code base.
>>
>> Thanks!
>>
>>
>>
>> On Sep 16, 2014, at 10:20 AM, Artem Polyakov > > wrote:
>>
>> > Hello,
>> >
>> > I would like to introduce OMPI timing framework that was included into
>> the trunk yesterday (r32738). The code is new so if you'll hit some bugs -
>> just let me know.
>> >
>> > The framework consists of the set of macro's and routines for internal
>> OMPI usage + standalone tool mpisync and few additional scripts:
>> mpirun_prof and ompi_timing_post. The set of features is very basic and I
>> am open for discussion of new things that are desirable there.
>> >
>> > To enable framework compilation you should configure OMPI with
>> --enable-timing option. If the option was passed to ./configure, standalone
>> tools and scripts will be installed into /bin.
>> >
>> > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a
>> set of macro's that should be used to preprocess out all mentions of the
>> timing code in case it w

Re: [OMPI devel] OPAL timing framework

[Artem Polyakov]

Jeff,
I addressed your comments and it is in the trunk for 2 weeks already. I
thought that you are in rush with hosting migration, so I didn't show up.
Current solution is not complete. I plan to return to it to implement
continuous synchronisation as I will have the timeframe. However it should
be definitely usable as is and I would be glad if anybody will find it
useful. The framework is general and can be used to profile any subsystems.

2014-09-18 20:09 GMT+07:00 Jeff Squyres (jsquyres) :

> I have a few comments:
>
> - This looks nice.  Thanks for the contribution.
>
> - I notice that the ORTE timing stuff is now a compile-time decision, not
> a run-time decision.  Do we care that we've now taken away the ability for
> users to do timings in a production build?
>
> - "clksync" -- can we use "clocksync"?  It's only 2 letters.  We tend to
> use real words in the OMPI code base; unnecessary abbreviation should be
> avoided.
>
> - r32738 introduced a several files into the code base that have no
> copyrights, and do not have the standard OMPI copyright header block.
> Please fix.
>
> - There's no documentation on how to use mpisync, mpirun_prof, or
> ompi_timing_post, even though they're installed when you --enable-timing.
> What are these 3 executables?  Can we get man pages?
>
> - What's the purpose of the MCA param orte_rml_base_timing?  A *quick*
> look through the code seems to indicate that it is ignored.
>
> - What's the purpose of the MCA params opal_clksync_file,
> opal_timing_file, and opal_timing_overhead?  E.g., what is a "clksync"
> file, what is it for, and what is its format?  Does the user have to
> provide one?  If so, how to you get one?  Or is it an output file?
> ...etc.  The brief descriptions given in the MCA help strings don't really
> provide enough information for someone who has no idea what the timing
> stuff is.  Also, can those 3 params have a common prefix?  I.e., it's not
> obvious that opal_clksync_file is related to opal_timing_* at all.
>
> - A *quick* look at ompi/tools/mpisync shows that a bunch of that code
> came from an external project.  Is the license compatible with OMPI's
> license?  What do we need to do to conform to their license?
>
> - opal/util/timings.h is protected by OPAL_SYS_TIMING_H -- shouldn't it be
> OPAL_UTIL_TIMINGS_H?
>
> - There's commented-out code in opal/util/timings.h.
>
> - There's no doxygen-style documentation in opal/util/timings.h to tell
> developers how to use it.
>
> - There's "TODO" comments in opal/util/timings.c; should those be fixed?
>
> - opal_config.h should be the first include in opal/util/timings.c.
>
> - If timing support is not to be compiled in, then opal/util/timings.c
> should not be be compiled via the Makefile.am (rather than entirely #if'ed
> out).
>
> It looks like this work is about 95% complete.  Finishing the remaining 5%
> would make it great and genuinely useful to the rest of the code base.
>
> Thanks!
>
>
>
> On Sep 16, 2014, at 10:20 AM, Artem Polyakov  wrote:
>
> > Hello,
> >
> > I would like to introduce OMPI timing framework that was included into
> the trunk yesterday (r32738). The code is new so if you'll hit some bugs -
> just let me know.
> >
> > The framework consists of the set of macro's and routines for internal
> OMPI usage + standalone tool mpisync and few additional scripts:
> mpirun_prof and ompi_timing_post. The set of features is very basic and I
> am open for discussion of new things that are desirable there.
> >
> > To enable framework compilation you should configure OMPI with
> --enable-timing option. If the option was passed to ./configure, standalone
> tools and scripts will be installed into /bin.
> >
> > The timing code is located in OPAL (opal/utils/timing.[ch]). There is a
> set of macro's that should be used to preprocess out all mentions of the
> timing code in case it wasn't requested with --enable-timing:
> > OPAL_TIMING_DECLARE(t) - declare timing handler structure with name "t".
> > OPAL_TIMING_DECLARE_EXT(x, t) - external declaration of a timing handler
> "t".
> > OPAL_TIMING_INIT(t) - initialize timing handler "t"
> > OPAL_TIMING_EVENT(x) - printf-like event declaration similar to
> OPAL_OUTPUT.
> > The information about the event will be quickly inserted into the linked
> list. Maximum event description is limited by OPAL_TIMING_DESCR_MAX.
> > The malloc is performed in buckets (OPAL_TIMING_BUFSIZE at once) and
> overhead (time to malloc and prepare the bucket) is accounted in
> corresponding list

[OMPI devel] OMPI BCOL hang with PMI1

[Artem Polyakov]

Hello, I have troubles with latest trunk if I use PMI1.

For example, if I use 2 nodes the application hangs. See backtraces from
both nodes below. From them I can see that second (non launching) node
hangs in bcol component selection. Here is the default setting of
bcol_base_string parameter:
bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll,ugni"
according to ompi_info. I don't know if it is correct that basesmuma is
duplicated or not.

Experiments with this parameter showed that it directly influences the bug:
export OMPI_MCA_bcol_base_string="" #  [SEGFAULT]
export OMPI_MCA_bcol_base_string="ptpcoll" #  [OK]
export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll" #  [OK]
export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload" #  [OK]
export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload,ugni" #  [OK]
export
OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll,iboffload,ugni" #
 [HANG]
export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll" #
[HANG]
export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload" # [OK]
export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ugni" # [OK]
export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll" #  [HANG]
export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma" #  [OK]
export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma,basesmuma" #  [HANG]

I can provide other information if nessesary.

cn1:
(gdb) bt
0  0x7fdebd30ac6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
1  0x7fdebcca64e0 in poll_dispatch (base=0x1d466b0, tv=0x7fff71aab880)
at poll.c:165
2  0x7fdebcc9b041 in opal_libevent2021_event_base_loop (base=0x1d466b0,
flags=2) at event.c:1631
3  0x7fdebcc35891 in opal_progress () at runtime/opal_progress.c:169
4  0x7fdeb32f78cb in opal_condition_wait (c=0x7fdebdb51bc0
, m=0x7fdebdb51cc0 ) at
../../../../opal/threads/condition.h:78
5  0x7fdeb32f79b8 in ompi_request_wait_completion (req=0x7fff71aab920)
at ../../../../ompi/request/request.h:381
6  0x7fdeb32f84b8 in mca_pml_ob1_recv (addr=0x7fff71aabd80, count=1,
datatype=0x6026c0 , src=1, tag=0, comm=0x6020a0
,
status=0x7fff71aabd90) at pml_ob1_irecv.c:109
7  0x7fdebd88f54d in PMPI_Recv (buf=0x7fff71aabd80, count=1,
type=0x6026c0 , source=1, tag=0, comm=0x6020a0
,
status=0x7fff71aabd90) at precv.c:78
8  0x00400c44 in main (argc=1, argv=0x7fff71aabe98) at hellompi.c:33

cn2:
(gdb) bt
0  0x7fa65aa78c6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
1  0x7fa65a4144e0 in poll_dispatch (base=0x20e96b0, tv=0x7fff46f44a80)
at poll.c:165
2  0x7fa65a409041 in opal_libevent2021_event_base_loop (base=0x20e96b0,
flags=2) at event.c:1631
3  0x7fa65a3a3891 in opal_progress () at runtime/opal_progress.c:169
4  0x7fa65afbbc25 in opal_condition_wait (c=0x7fa65b2bfbc0
, m=0x7fa65b2bfcc0 ) at
../opal/threads/condition.h:78
5  0x7fa65afbc1b5 in ompi_request_default_wait_all (count=2,
requests=0x7fff46f44c70, statuses=0x0) at request/req_wait.c:287
6  0x7fa65afc7906 in comm_allgather_pml (src_buf=0x7fff46f44da0,
dest_buf=0x233dac0, count=288, dtype=0x7fa65b29fee0 ,
my_rank_in_group=1,
n_peers=2, ranks_in_comm=0x210a760, comm=0x6020a0
) at patterns/comm/allgather.c:250
7  0x7fa64f14ba08 in bcol_basesmuma_smcm_allgather_connection
(sm_bcol_module=0x7fa64e64d010, module=0x232c800,
peer_list=0x7fa64f3513e8 ,
back_files=0x7fa64eae2690, comm=0x6020a0 , input=...,
base_fname=0x7fa64f14ca8c "sm_ctl_mem_", map_all=false) at
bcol_basesmuma_smcm.c:205
8  0x7fa64f146525 in base_bcol_basesmuma_setup_ctl
(sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220
) at bcol_basesmuma_setup.c:344
9  0x7fa64f146cbb in base_bcol_basesmuma_setup_library_buffers
(sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220
)
at bcol_basesmuma_setup.c:550
10 0x7fa64f1418d0 in mca_bcol_basesmuma_comm_query (module=0x232c800,
num_modules=0x232e570) at bcol_basesmuma_module.c:532
11 0x7fa64fd9e5f2 in mca_coll_ml_tree_hierarchy_discovery
(ml_module=0x232fbe0, topo=0x232fd98, n_hierarchies=3,
exclude_sbgp_name=0x0, include_sbgp_name=0x0)
at coll_ml_module.c:1964
12 0x7fa64fd9f3a3 in mca_coll_ml_fulltree_hierarchy_discovery
(ml_module=0x232fbe0, n_hierarchies=3) at coll_ml_module.c:2211
13 0x7fa64fd9cbe4 in ml_discover_hierarchy (ml_module=0x232fbe0) at
coll_ml_module.c:1518
14 0x7fa64fda164f in mca_coll_ml_comm_query (comm=0x6020a0
, priority=0x7fff46f45358) at coll_ml_module.c:2970
15 0x7fa65b02f6aa in query_2_0_0 (component=0x7fa64fffe4e0
, comm=0x6020a0 ,
priority=0x7fff46f45358,
module=0x7fff46f45390) at base/coll_base_comm_select.c:374
16 0x7fa65b02f66e in query (component=0x7fa64fffe4e0
, comm=0x6020a0 ,
priority=0x7fff46f45358, module=0x7fff46f45390)
at base/coll_base_comm_select.c:357
17 0x7fa65b02f581 in check_one_component (comm=0x6020a0
, component=0x7fa64fffe4e0 ,
module=0x7fff46f45390)
at base/coll_base_comm_select.c:319
18 0x7fa65b02f3c7 in check_components (c

Re: [OMPI devel] OMPI BCOL hang with PMI1

[Artem Polyakov]

Gilles,

I checked your patch and it doesn't solve the problem I observe. I think
the reason is somewhere else.

2014-10-17 19:13 GMT+07:00 Gilles Gouaillardet <
gilles.gouaillar...@gmail.com>:

> Artem,
>
> There is a known issue #235 with modex and i made PR #238 with a tentative
> fix.
>
> Could you please give it a try and reports if it solves your problem ?
>
> Cheers
>
> Gilles
>
>
> Artem Polyakov  wrote:
> Hello, I have troubles with latest trunk if I use PMI1.
>
> For example, if I use 2 nodes the application hangs. See backtraces from
> both nodes below. From them I can see that second (non launching) node
> hangs in bcol component selection. Here is the default setting of
> bcol_base_string parameter:
> bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll,ugni"
> according to ompi_info. I don't know if it is correct that basesmuma is
> duplicated or not.
>
> Experiments with this parameter showed that it directly influences the bug:
> export OMPI_MCA_bcol_base_string="" #  [SEGFAULT]
> export OMPI_MCA_bcol_base_string="ptpcoll" #  [OK]
> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll" #  [OK]
> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload" #  [OK]
> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload,ugni" #  [OK]
> export
> OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll,iboffload,ugni" #
>  [HANG]
> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll" #
> [HANG]
> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload" # [OK]
> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ugni" #
> [OK]
> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll" #  [HANG]
> export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma" #  [OK]
> export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma,basesmuma" #  [HANG]
>
> I can provide other information if nessesary.
>
> cn1:
> (gdb) bt
> 0  0x7fdebd30ac6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
> 1  0x7fdebcca64e0 in poll_dispatch (base=0x1d466b0, tv=0x7fff71aab880)
> at poll.c:165
> 2  0x7fdebcc9b041 in opal_libevent2021_event_base_loop
> (base=0x1d466b0, flags=2) at event.c:1631
> 3  0x7fdebcc35891 in opal_progress () at runtime/opal_progress.c:169
> 4  0x7fdeb32f78cb in opal_condition_wait (c=0x7fdebdb51bc0
> , m=0x7fdebdb51cc0 ) at
> ../../../../opal/threads/condition.h:78
> 5  0x7fdeb32f79b8 in ompi_request_wait_completion (req=0x7fff71aab920)
> at ../../../../ompi/request/request.h:381
> 6  0x7fdeb32f84b8 in mca_pml_ob1_recv (addr=0x7fff71aabd80, count=1,
> datatype=0x6026c0 , src=1, tag=0, comm=0x6020a0
> ,
> status=0x7fff71aabd90) at pml_ob1_irecv.c:109
> 7  0x7fdebd88f54d in PMPI_Recv (buf=0x7fff71aabd80, count=1,
> type=0x6026c0 , source=1, tag=0, comm=0x6020a0
> ,
> status=0x7fff71aabd90) at precv.c:78
> 8  0x00400c44 in main (argc=1, argv=0x7fff71aabe98) at
> hellompi.c:33
>
> cn2:
> (gdb) bt
> 0  0x7fa65aa78c6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
> 1  0x7fa65a4144e0 in poll_dispatch (base=0x20e96b0, tv=0x7fff46f44a80)
> at poll.c:165
> 2  0x7fa65a409041 in opal_libevent2021_event_base_loop
> (base=0x20e96b0, flags=2) at event.c:1631
> 3  0x7fa65a3a3891 in opal_progress () at runtime/opal_progress.c:169
> 4  0x7fa65afbbc25 in opal_condition_wait (c=0x7fa65b2bfbc0
> , m=0x7fa65b2bfcc0 ) at
> ../opal/threads/condition.h:78
> 5  0x7fa65afbc1b5 in ompi_request_default_wait_all (count=2,
> requests=0x7fff46f44c70, statuses=0x0) at request/req_wait.c:287
> 6  0x7fa65afc7906 in comm_allgather_pml (src_buf=0x7fff46f44da0,
> dest_buf=0x233dac0, count=288, dtype=0x7fa65b29fee0 ,
> my_rank_in_group=1,
> n_peers=2, ranks_in_comm=0x210a760, comm=0x6020a0
> ) at patterns/comm/allgather.c:250
> 7  0x7fa64f14ba08 in bcol_basesmuma_smcm_allgather_connection
> (sm_bcol_module=0x7fa64e64d010, module=0x232c800,
> peer_list=0x7fa64f3513e8 ,
> back_files=0x7fa64eae2690, comm=0x6020a0 , input=...,
> base_fname=0x7fa64f14ca8c "sm_ctl_mem_", map_all=false) at
> bcol_basesmuma_smcm.c:205
> 8  0x7fa64f146525 in base_bcol_basesmuma_setup_ctl
> (sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220
> ) at bcol_basesmuma_setup.c:344
> 9  0x7fa64f146cbb in base_bcol_basesmuma_setup_library_buffers
> (sm_bcol_module=0x7fa64e64d010, cs=0x7fa64f351220
> )
> at bcol_basesmuma_setup.c:550
> 10 0x7fa64f1418d0 in mca_bcol_basesmuma_comm_query (module=0x232c800,
> num_modules=0x232e570) at bcol_basesmuma_module.c:532
&g

Re: [OMPI devel] OMPI BCOL hang with PMI1

[Artem Polyakov]

Hey, Lena :).

2014-10-17 22:07 GMT+07:00 Elena Elkina :

> Hi Artem,
>
> Actually some time ago there was a known issue with coll ml. I used to run
> my command lines with -mca coll ^ml to avoid these problems, so I don't
> know if it was fixed or not. It looks like you have the same problem.
>

but mine is with bcol, not coll framework. And as you can see modules
itself doesn't brake the program. Only some of their combinations. Also I
am curious why basesmuma module listed twice.



> Best regards,
> Elena
>
> On Fri, Oct 17, 2014 at 7:01 PM, Artem Polyakov 
> wrote:
>
>> Gilles,
>>
>> I checked your patch and it doesn't solve the problem I observe. I think
>> the reason is somewhere else.
>>
>> 2014-10-17 19:13 GMT+07:00 Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com>:
>>
>>> Artem,
>>>
>>> There is a known issue #235 with modex and i made PR #238 with a
>>> tentative fix.
>>>
>>> Could you please give it a try and reports if it solves your problem ?
>>>
>>> Cheers
>>>
>>> Gilles
>>>
>>>
>>> Artem Polyakov  wrote:
>>> Hello, I have troubles with latest trunk if I use PMI1.
>>>
>>> For example, if I use 2 nodes the application hangs. See backtraces from
>>> both nodes below. From them I can see that second (non launching) node
>>> hangs in bcol component selection. Here is the default setting of
>>> bcol_base_string parameter:
>>> bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll,ugni"
>>> according to ompi_info. I don't know if it is correct that basesmuma is
>>> duplicated or not.
>>>
>>> Experiments with this parameter showed that it directly influences the
>>> bug:
>>> export OMPI_MCA_bcol_base_string="" #  [SEGFAULT]
>>> export OMPI_MCA_bcol_base_string="ptpcoll" #  [OK]
>>> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll" #  [OK]
>>> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload" #  [OK]
>>> export OMPI_MCA_bcol_base_string="basesmuma,ptpcoll,iboffload,ugni" #
>>>  [OK]
>>> export
>>> OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll,iboffload,ugni" #
>>>  [HANG]
>>> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ptpcoll"
>>> # [HANG]
>>> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload" # [OK]
>>> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,iboffload,ugni" #
>>> [OK]
>>> export OMPI_MCA_bcol_base_string="basesmuma,basesmuma,ptpcoll" #  [HANG]
>>> export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma" #  [OK]
>>> export OMPI_MCA_bcol_base_string="ptpcoll,basesmuma,basesmuma" #  [HANG]
>>>
>>> I can provide other information if nessesary.
>>>
>>> cn1:
>>> (gdb) bt
>>> 0  0x7fdebd30ac6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
>>> 1  0x7fdebcca64e0 in poll_dispatch (base=0x1d466b0,
>>> tv=0x7fff71aab880) at poll.c:165
>>> 2  0x7fdebcc9b041 in opal_libevent2021_event_base_loop
>>> (base=0x1d466b0, flags=2) at event.c:1631
>>> 3  0x7fdebcc35891 in opal_progress () at runtime/opal_progress.c:169
>>> 4  0x7fdeb32f78cb in opal_condition_wait (c=0x7fdebdb51bc0
>>> , m=0x7fdebdb51cc0 ) at
>>> ../../../../opal/threads/condition.h:78
>>> 5  0x7fdeb32f79b8 in ompi_request_wait_completion
>>> (req=0x7fff71aab920) at ../../../../ompi/request/request.h:381
>>> 6  0x7fdeb32f84b8 in mca_pml_ob1_recv (addr=0x7fff71aabd80, count=1,
>>> datatype=0x6026c0 , src=1, tag=0, comm=0x6020a0
>>> ,
>>> status=0x7fff71aabd90) at pml_ob1_irecv.c:109
>>> 7  0x7fdebd88f54d in PMPI_Recv (buf=0x7fff71aabd80, count=1,
>>> type=0x6026c0 , source=1, tag=0, comm=0x6020a0
>>> ,
>>> status=0x7fff71aabd90) at precv.c:78
>>> 8  0x00400c44 in main (argc=1, argv=0x7fff71aabe98) at
>>> hellompi.c:33
>>>
>>> cn2:
>>> (gdb) bt
>>> 0  0x7fa65aa78c6d in poll () from /lib/x86_64-linux-gnu/libc.so.6
>>> 1  0x7fa65a4144e0 in poll_dispatch (base=0x20e96b0,
>>> tv=0x7fff46f44a80) at poll.c:165
>>> 2  0x7fa65a409041 in opal_libevent2021_event_base_loop
>>> (base=0x20e96b0, flags=2) at event.c:1631
>>> 3  0x7fa65a3a3891 in opal_progress () at runtime/opal_progress.c:169
>>> 

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

I think this might be related to the configuration problem I was fixing
with Jeff few months ago. Refer here:
https://github.com/open-mpi/ompi/pull/240

2014-12-02 10:15 GMT+06:00 Ralph Castain :

> If it isn’t too much trouble, it would be good to confirm that it remains
> broken. I strongly suspect it is based on Moe’s comments.
>
> Obviously, other people are making this work. For Intel MPI, all you do is
> point it at libpmi and they can run. However, they do explicitly dlopen it
> in their code, and I don’t know what flags they might pass when they do so.
>
> If necessary, I suppose we could follow that pattern. In other words,
> rather than specifically linking the “s1” component to libpmi, instead
> require that the user point us to a pmi library via an MCA param, then
> explicitly dlopen that library with RTLD_GLOBAL. This avoids the issues
> cited by Jeff, but resolves the pmi linkage problem.
>
>
> On Dec 1, 2014, at 8:09 PM, Gilles Gouaillardet <
> gilles.gouaillar...@iferc.org> wrote:
>
>  $ srun --version
> slurm 2.6.6-VENDOR_PROVIDED
>
> $ srun --mpi=pmi2 -n 1 ~/hw
> I am 0 / 1
>
> $ srun -n 1 ~/hw
> /csc/home1/gouaillardet/hw: symbol lookup error:
> /usr/lib64/slurm/auth_munge.so: undefined symbol: slurm_verbose
> srun: error: slurm_receive_msg: Zero Bytes were transmitted or received
> srun: error: slurm_receive_msg[10.0.3.15]: Zero Bytes were transmitted or
> received
> srun: error: soleil: task 0: Exited with exit code 127
>
> $ ldd /usr/lib64/slurm/auth_munge.so
> linux-vdso.so.1 =>  (0x7fff54478000)
> libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x7f744760f000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x7f74473f1000)
> libc.so.6 => /lib64/libc.so.6 (0x7f744705d000)
> /lib64/ld-linux-x86-64.so.2 (0x003bf540)
>
>
> now, if i reling auth_munge.so so it depends on libslurm :
>
> $ srun -n 1 ~/hw
> srun: symbol lookup error: /usr/lib64/slurm/auth_munge.so: undefined
> symbol: slurm_auth_get_arg_desc
>
>
> i can give a try to the latest slurm if needed
>
> Cheers,
>
> Gilles
>
>
> On 2014/12/02 12:56, Ralph Castain wrote:
>
> Out of curiosity - how are you testing these? I have more current versions of 
> Slurm and would like to test the observations there.
>
>
>  On Dec 1, 2014, at 7:49 PM, Gilles Gouaillardet 
>   wrote:
>
> I d like to make a step back ...
>
> i previously tested with slurm 2.6.0, and it complained about the 
> slurm_verbose symbol that is defined in libslurm.so
> so with slurm 2.6.0, RTLD_GLOBAL or relinking is ok
>
> now i tested with slurm 2.6.6 and it complains about the 
> slurm_auth_get_arg_desc symbol, and this symbol is not
> defined in any dynamic library. it is internally defined in the static 
> libcommon.a library, which is used to build the slurm binaries.
>
> as far as i understand, auth_munge.so can only be invoked from a slurm 
> binary, which means it cannot be invoked from an mpi application
> even if it is linked with libslurm, libpmi, ...
>
> that looks like a slurm design issue that the slurm folks will take care of.
>
> Cheers,
>
> Gilles
>
> On 2014/12/02 12:33, Ralph Castain wrote:
>
>  Another option is to simply add the -lslurm -lauth flags to the pmix/s1 
> component as this is the only place that requires it, and it won’t hurt 
> anything to do so.
>
>
>
>  On Dec 1, 2014, at 6:03 PM, Gilles Gouaillardet 
>   
>   wrote:
>
> Jeff,
>
> FWIW, you can read my analysis of what is going wrong 
> athttp://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php 
>  
>  
>  
>  
>  
> 
>
> bottom line, i agree this is a slurm issue (slurm plugin should depend
> on libslurm, but they do not, yet)
>
> a possible workaround would be to make the pmi component a "proxy" that
> dlopen with RTLD_GLOBAL the "real" component in which the job is done.
> that being said, the impact is quite limited (no direct launch in slurm
> with pmi1, but pmi2 works fine) so it makes sense not to work around
> someone else problem.
> and that being said, configure could detect this broken pmi1 and not
> build pmi1 support or print a user friendly error message if pmi1 is used.
>
> any thoughts ?
>
> Cheers,
>
> Gilles
>
> On 2014/12/02 7:47, Jeff Squyres (jsquyres) wrote:
>
>  Ok, if the problem is moot, great.
>
> (sidenote: this is moot, so ignore this if you want: with this explanation, 
> I'm still not sure how RTLD_GLOBAL fixes the issue)
>
>
> On Dec 1, 2014, at 5:15 PM, Ralph Castain  
>    wrote:
>
>
>  Easy enough to explain. We link libpmi into the 

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

2014-12-02 17:13 GMT+06:00 Ralph Castain :

> Hmmm…if that is true, then it didn’t fix this problem as it is being
> reported in the master.
>

I had this problem on my laptop installation. You can check my report it
was detailed enough and see if you hitting the same issue. My fix was also
included into 1.8 branch. I am not sure that this is the same issue but
they looks similar.


>
>
> On Dec 1, 2014, at 9:40 PM, Artem Polyakov  wrote:
>
> I think this might be related to the configuration problem I was fixing
> with Jeff few months ago. Refer here:
> https://github.com/open-mpi/ompi/pull/240
>
> 2014-12-02 10:15 GMT+06:00 Ralph Castain :
>
>> If it isn’t too much trouble, it would be good to confirm that it remains
>> broken. I strongly suspect it is based on Moe’s comments.
>>
>> Obviously, other people are making this work. For Intel MPI, all you do
>> is point it at libpmi and they can run. However, they do explicitly dlopen
>> it in their code, and I don’t know what flags they might pass when they do
>> so.
>>
>> If necessary, I suppose we could follow that pattern. In other words,
>> rather than specifically linking the “s1” component to libpmi, instead
>> require that the user point us to a pmi library via an MCA param, then
>> explicitly dlopen that library with RTLD_GLOBAL. This avoids the issues
>> cited by Jeff, but resolves the pmi linkage problem.
>>
>>
>> On Dec 1, 2014, at 8:09 PM, Gilles Gouaillardet <
>> gilles.gouaillar...@iferc.org> wrote:
>>
>>  $ srun --version
>> slurm 2.6.6-VENDOR_PROVIDED
>>
>> $ srun --mpi=pmi2 -n 1 ~/hw
>> I am 0 / 1
>>
>> $ srun -n 1 ~/hw
>> /csc/home1/gouaillardet/hw: symbol lookup error:
>> /usr/lib64/slurm/auth_munge.so: undefined symbol: slurm_verbose
>> srun: error: slurm_receive_msg: Zero Bytes were transmitted or received
>> srun: error: slurm_receive_msg[10.0.3.15]: Zero Bytes were transmitted or
>> received
>> srun: error: soleil: task 0: Exited with exit code 127
>>
>> $ ldd /usr/lib64/slurm/auth_munge.so
>> linux-vdso.so.1 =>  (0x7fff54478000)
>> libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x7f744760f000)
>> libpthread.so.0 => /lib64/libpthread.so.0 (0x7f74473f1000)
>> libc.so.6 => /lib64/libc.so.6 (0x7f744705d000)
>> /lib64/ld-linux-x86-64.so.2 (0x003bf540)
>>
>>
>> now, if i reling auth_munge.so so it depends on libslurm :
>>
>> $ srun -n 1 ~/hw
>> srun: symbol lookup error: /usr/lib64/slurm/auth_munge.so: undefined
>> symbol: slurm_auth_get_arg_desc
>>
>>
>> i can give a try to the latest slurm if needed
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/02 12:56, Ralph Castain wrote:
>>
>> Out of curiosity - how are you testing these? I have more current versions 
>> of Slurm and would like to test the observations there.
>>
>>
>>  On Dec 1, 2014, at 7:49 PM, Gilles Gouaillardet 
>>   wrote:
>>
>> I d like to make a step back ...
>>
>> i previously tested with slurm 2.6.0, and it complained about the 
>> slurm_verbose symbol that is defined in libslurm.so
>> so with slurm 2.6.0, RTLD_GLOBAL or relinking is ok
>>
>> now i tested with slurm 2.6.6 and it complains about the 
>> slurm_auth_get_arg_desc symbol, and this symbol is not
>> defined in any dynamic library. it is internally defined in the static 
>> libcommon.a library, which is used to build the slurm binaries.
>>
>> as far as i understand, auth_munge.so can only be invoked from a slurm 
>> binary, which means it cannot be invoked from an mpi application
>> even if it is linked with libslurm, libpmi, ...
>>
>> that looks like a slurm design issue that the slurm folks will take care of.
>>
>> Cheers,
>>
>> Gilles
>>
>> On 2014/12/02 12:33, Ralph Castain wrote:
>>
>>  Another option is to simply add the -lslurm -lauth flags to the pmix/s1 
>> component as this is the only place that requires it, and it won’t hurt 
>> anything to do so.
>>
>>
>>
>>  On Dec 1, 2014, at 6:03 PM, Gilles Gouaillardet 
>>   
>> <mailto:gilles.gouaillar...@iferc.org>  wrote:
>>
>> Jeff,
>>
>> FWIW, you can read my analysis of what is going wrong 
>> athttp://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php 
>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php> 
>> <http://www.open-mpi.org/community/lists/pmix-devel/2014/11/0293.php> 
>> <http://www.open-mpi.org/

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

Agree. First you should check is to what value OPAL_HAVE_LTDL_ADVISE is
set. If it is zero - very probably this is the same bug as mine.

2014-12-02 17:33 GMT+06:00 Ralph Castain :

> It does look similar - question is: why didn’t this fix the problem? Will
> have to investigate.
>
> Thanks
>
>
> On Dec 2, 2014, at 3:17 AM, Artem Polyakov  wrote:
>
>
>
> 2014-12-02 17:13 GMT+06:00 Ralph Castain :
>
>> Hmmm…if that is true, then it didn’t fix this problem as it is being
>> reported in the master.
>>
>
> I had this problem on my laptop installation. You can check my report it
> was detailed enough and see if you hitting the same issue. My fix was also
> included into 1.8 branch. I am not sure that this is the same issue but
> they looks similar.
>
>
>>
>>
>> On Dec 1, 2014, at 9:40 PM, Artem Polyakov  wrote:
>>
>> I think this might be related to the configuration problem I was fixing
>> with Jeff few months ago. Refer here:
>> https://github.com/open-mpi/ompi/pull/240
>>
>> 2014-12-02 10:15 GMT+06:00 Ralph Castain :
>>
>>> If it isn’t too much trouble, it would be good to confirm that it
>>> remains broken. I strongly suspect it is based on Moe’s comments.
>>>
>>> Obviously, other people are making this work. For Intel MPI, all you do
>>> is point it at libpmi and they can run. However, they do explicitly dlopen
>>> it in their code, and I don’t know what flags they might pass when they do
>>> so.
>>>
>>> If necessary, I suppose we could follow that pattern. In other words,
>>> rather than specifically linking the “s1” component to libpmi, instead
>>> require that the user point us to a pmi library via an MCA param, then
>>> explicitly dlopen that library with RTLD_GLOBAL. This avoids the issues
>>> cited by Jeff, but resolves the pmi linkage problem.
>>>
>>>
>>> On Dec 1, 2014, at 8:09 PM, Gilles Gouaillardet <
>>> gilles.gouaillar...@iferc.org> wrote:
>>>
>>> $ srun --version
>>> slurm 2.6.6-VENDOR_PROVIDED
>>>
>>> $ srun --mpi=pmi2 -n 1 ~/hw
>>> I am 0 / 1
>>>
>>> $ srun -n 1 ~/hw
>>> /csc/home1/gouaillardet/hw: symbol lookup error:
>>> /usr/lib64/slurm/auth_munge.so: undefined symbol: slurm_verbose
>>> srun: error: slurm_receive_msg: Zero Bytes were transmitted or received
>>> srun: error: slurm_receive_msg[10.0.3.15]: Zero Bytes were transmitted
>>> or received
>>> srun: error: soleil: task 0: Exited with exit code 127
>>>
>>> $ ldd /usr/lib64/slurm/auth_munge.so
>>> linux-vdso.so.1 =>  (0x7fff54478000)
>>> libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x7f744760f000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x7f74473f1000)
>>> libc.so.6 => /lib64/libc.so.6 (0x7f744705d000)
>>> /lib64/ld-linux-x86-64.so.2 (0x003bf540)
>>>
>>>
>>> now, if i reling auth_munge.so so it depends on libslurm :
>>>
>>> $ srun -n 1 ~/hw
>>> srun: symbol lookup error: /usr/lib64/slurm/auth_munge.so: undefined
>>> symbol: slurm_auth_get_arg_desc
>>>
>>>
>>> i can give a try to the latest slurm if needed
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On 2014/12/02 12:56, Ralph Castain wrote:
>>>
>>> Out of curiosity - how are you testing these? I have more current versions 
>>> of Slurm and would like to test the observations there.
>>>
>>>
>>> On Dec 1, 2014, at 7:49 PM, Gilles Gouaillardet 
>>>   wrote:
>>>
>>> I d like to make a step back ...
>>>
>>> i previously tested with slurm 2.6.0, and it complained about the 
>>> slurm_verbose symbol that is defined in libslurm.so
>>> so with slurm 2.6.0, RTLD_GLOBAL or relinking is ok
>>>
>>> now i tested with slurm 2.6.6 and it complains about the 
>>> slurm_auth_get_arg_desc symbol, and this symbol is not
>>> defined in any dynamic library. it is internally defined in the static 
>>> libcommon.a library, which is used to build the slurm binaries.
>>>
>>> as far as i understand, auth_munge.so can only be invoked from a slurm 
>>> binary, which means it cannot be invoked from an mpi application
>>> even if it is linked with libslurm, libpmi, ...
>>>
>>> that looks like a slurm design issue that the slurm folks will take care of.
>>>
>>> Cheers,
>>>
>>> Gi

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

2014-12-02 20:59 GMT+06:00 Edgar Gabriel :

> didn't want to interfere with this thread, although I have a similar
> issue, since I have the solution nearly fully cooked up. But anyway, this
> last email gave the hint on why we have suddenly the problem in ompio:
>
> it looks like OPAL_HAVE_LTDL_ADVISE (at least on my systems) is not set
> anymore, so the entire section is being skipped. I double checked that with
> the 1.8 branch, it goes through the section, but not with master.
>

Hi, Edgar.

Both master and ompi-release (isn't it 1.8?!) are equal in sence of my fix.
Something else!? I'd like to see config.log too but will look into it only
tomorrow.

Also I want to add that SLURM PMI2 communicates with local slurmstepd's and
doesn't need any authentification. All PMI1 processes otherwise communicate
to the srun process and thus need libslurm services for communication and
authentification.


>
> Thanks
> Edgar
>
>
>
>
> On 12/2/2014 7:56 AM, Jeff Squyres (jsquyres) wrote:
>
>> Looks like I was totally lying in http://www.open-mpi.org/
>> community/lists/devel/2014/12/16381.php (where I said we should not use
>> RTLD_GLOBAL).  We *do* use RTLD_GLOBAL:
>>
>> https://github.com/open-mpi/ompi/blob/master/opal/mca/
>> base/mca_base_component_repository.c#L124
>>
>> This ltdl advice object is passed to lt_dlopen() for all components.  My
>> mistake; sorry.
>>
>> So the idea that using RTLD_GLOBAL will fix this SLURM bug is incorrect.
>>
>> I believe someone said earlier in the thread that adding the right -llibs
>> to the configure line will solve the issue, and that sounds correct to me.
>> If there's a missing symbol because the SLURM libraries are not
>> automatically pulling in the right dependent libraries, then *if* we put a
>> workaround in OMPI to fix this issue, then the right workaround is to add
>> the relevant -llibs when that component is linked.
>>
>> *If* you add that workaround (which is a whole separate discussion), I
>> would suggest adding a configure.m4 test to see if adding the additional
>> -llibs are necessary.  Perhaps AC_LINK_IFELSE looking for a symbol, and
>> then if that fails, AC_LINK_IFELSE again with the additional -llibs to see
>> if that works.
>>
>> Or something like that.
>>
>>
>>
>> On Dec 2, 2014, at 6:38 AM, Artem Polyakov  wrote:
>>
>>  Agree. First you should check is to what value OPAL_HAVE_LTDL_ADVISE is
>>> set. If it is zero - very probably this is the same bug as mine.
>>>
>>> 2014-12-02 17:33 GMT+06:00 Ralph Castain :
>>> It does look similar - question is: why didn’t this fix the problem?
>>> Will have to investigate.
>>>
>>> Thanks
>>>
>>>
>>>  On Dec 2, 2014, at 3:17 AM, Artem Polyakov  wrote:
>>>>
>>>>
>>>>
>>>> 2014-12-02 17:13 GMT+06:00 Ralph Castain :
>>>> Hmmm…if that is true, then it didn’t fix this problem as it is being
>>>> reported in the master.
>>>>
>>>> I had this problem on my laptop installation. You can check my report
>>>> it was detailed enough and see if you hitting the same issue. My fix was
>>>> also included into 1.8 branch. I am not sure that this is the same issue
>>>> but they looks similar.
>>>>
>>>>
>>>>
>>>>  On Dec 1, 2014, at 9:40 PM, Artem Polyakov  wrote:
>>>>>
>>>>> I think this might be related to the configuration problem I was
>>>>> fixing with Jeff few months ago. Refer here:
>>>>> https://github.com/open-mpi/ompi/pull/240
>>>>>
>>>>> 2014-12-02 10:15 GMT+06:00 Ralph Castain :
>>>>> If it isn’t too much trouble, it would be good to confirm that it
>>>>> remains broken. I strongly suspect it is based on Moe’s comments.
>>>>>
>>>>> Obviously, other people are making this work. For Intel MPI, all you
>>>>> do is point it at libpmi and they can run. However, they do explicitly
>>>>> dlopen it in their code, and I don’t know what flags they might pass when
>>>>> they do so.
>>>>>
>>>>> If necessary, I suppose we could follow that pattern. In other words,
>>>>> rather than specifically linking the “s1” component to libpmi, instead
>>>>> require that the user point us to a pmi library via an MCA param, then
>>>>> explicitly dlopen that library with RTLD_GLOBAL. This avoids the issues
>>>>> cited by Jeff, but resolv

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

Hello,

Jeff, your fix brakes my system again. Actually you just reverted my
changes. Here is what I have:

configure:5441: *** GNU libltdl setup
configure:296939: checking location of libltdl
configure:296952: result: internal copy
configure:297028: OPAL configuring in opal/libltdl
configure:297113: running /bin/bash '.../opal/libltdl/configure'
 '--prefix=.../ompi-pmix-refactoring_install/' '--enable-debug'
'--disable-oshmem' '--with-pmi=/home/artpol/sandboxes/slurm/'
--enable-ltdl-convenience --disable-ltdl-install --enable-shared
--disable-static --cache-file=/dev/null --srcdir=.../opal/libltdl
--disable-option-checking
configure:297119: /bin/bash '.../opal/libltdl/configure' succeeded for
opal/libltdl
In file included from conftest.c:718:0:
.../opal/libltdl/ltdl.h:36:31: fatal error: libltdl/lt_system.h: No such
file or directory
 #include 
   ^
compilation terminated.
configure:297864: checking for lt_dladvise
configure:297870: result: no
configure:297923: creating ./config.lt

Surprisingly to me this error (I am sure!) occurs on any system but only on
mine it fails to set advise on! I checked that on other machines!

The reason was pointed in original PR:
ltdl.h has includes

#include < libltdl/lt_system.h >
#include < libltdl/lt_error.h >


That can't be found without "-I$srcdir/opal/libltdl/".

The point is that we DO need "-I$srcdir/opal/libltdl/" but we ALSO need
"-I$srcdir" too! I filed the new PR (
https://github.com/open-mpi/ompi/pull/301) but won't merge it until Edgar
confirms that it's OK with his system.

So the original error was in removing -I$srcdir. I was sure that we
converged on this and found another valuable discussion in ompi-release:
https://github.com/open-mpi/ompi-release/pull/34

There I was looking into configure script and found:

CPPFLAGS="-I$srcdir/ -I$srcdir/opal/libltdl/"# Must specifically
mention $srcdir here for VPATH builds# (this file is in the src tree).
cat confdefs.h - <<_ACEOF >conftest.$ac_ext/* end confdefs.h.
*/#include <$srcdir/opal/libltdl/ltdl.h>_ACEOF


And it was obvious that we don't need "-I$srcdir/" because it was hardcoded
in the include but it turns out that I've been wrong and maybe some other
building system emmits different code. I would like to see Edgars original
config.log. Jeff could you send it to me directly?

So, everybody, sorry for inconvinience!


2014-12-03 0:41 GMT+06:00 Jeff Squyres (jsquyres) :

> See https://github.com/open-mpi/ompi/pull/298 for a fix.
>
> There's 2 commits on that PR -- the 2nd is just a cleanup.  The real fix
> is the 1st commit, here:
>
>
> https://github.com/jsquyres/ompi/commit/a736d83fb9a7b27986a008a2cda6eb1fea839fb3
>
> If someone can confirm that this works for them, we can bring it to master.
>
> It may have the side effect of "fixing / working around" (by coincidence)
> the SLURM bug (we all agree that the Right solution is to have SLURM fix it
> upstream, but I think this will put us back in the case of "working by
> accident / despite the SLURM bug").
>
>
>
> On Dec 2, 2014, at 10:59 AM, Jeff Squyres (jsquyres) 
> wrote:
>
> > I'm able to replicate Edgar's problem.
> >
> > I'm investigating...
> >
> >
> > On Dec 2, 2014, at 10:39 AM, Edgar Gabriel  wrote:
> >
> >> the mailing list refused to let me add the config.log file, since it is
> too large, I can forward the output to you directly as well (as I did to
> Jeff).
> >>
> >> I honestly have not looked into the configure logic, I can just tell
> that OPAL_HAVE_LTDL_ADVISE is not set on my linux system for master, but is
> set on the 1.8 series (1.8 series checkout was from Nov. 20, so if
> something changed in between the result might be different).
> >>
> >>
> >>
> >> On 12/2/2014 9:27 AM, Artem Polyakov wrote:
> >>>
> >>> 2014-12-02 20:59 GMT+06:00 Edgar Gabriel  >>> <mailto:gabr...@cs.uh.edu>>:
> >>>
> >>>   didn't want to interfere with this thread, although I have a similar
> >>>   issue, since I have the solution nearly fully cooked up. But anyway,
> >>>   this last email gave the hint on why we have suddenly the problem in
> >>>   ompio:
> >>>
> >>>   it looks like OPAL_HAVE_LTDL_ADVISE (at least on my systems) is not
> >>>   set anymore, so the entire section is being skipped. I double
> >>>   checked that with the 1.8 branch, it goes through the section, but
> >>>   not with master.
> >>>
> >>>
> >>> Hi, Edgar.
> >>>
> >&

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

2014-12-03 8:30 GMT+06:00 Jeff Squyres (jsquyres) :

> On Dec 2, 2014, at 8:43 PM, Artem Polyakov  wrote:
>
> > Jeff, your fix brakes my system again. Actually you just reverted my
> changes.
>
> No, I didn't just revert them -- I made changes.  I did forget about the
> second -I, though (to be fair, the 2nd -I was the *only* -I in there before
> I committed).
>
Yeah! I was speaking figurally :).


> Sorry about that -- I've tested your change (without the trailing /) and
> it seems to work ok.  I'd go ahead and merge.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16414.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

Howard, does current mater fix your problems?

среда, 3 декабря 2014 г. пользователь Artem Polyakov написал:

>
> 2014-12-03 8:30 GMT+06:00 Jeff Squyres (jsquyres)  >:
>
>> On Dec 2, 2014, at 8:43 PM, Artem Polyakov > > wrote:
>>
>> > Jeff, your fix brakes my system again. Actually you just reverted my
>> changes.
>>
>> No, I didn't just revert them -- I made changes.  I did forget about the
>> second -I, though (to be fair, the 2nd -I was the *only* -I in there before
>> I committed).
>>
> Yeah! I was speaking figurally :).
>
>
>> Sorry about that -- I've tested your change (without the trailing /) and
>> it seems to work ok.  I'd go ahead and merge.
>>
>> --
>> Jeff Squyres
>> jsquy...@cisco.com 
>> For corporate legal information go to:
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org 
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2014/12/16414.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>


-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

I finally found the clear reason of this strange situation!

In ompi opal_setup_libltdl.m4 has the following content:
CPPFLAGS="-I$srcdir -I$srcdir/opal/libltdl"
AC_EGREP_HEADER([lt_dladvise_init], [opal/libltdl/ltdl.h],
[OPAL_HAVE_LTDL_ADVISE=1])

And in ompi-release opal_setup_libltdl.m4:
CPPFLAGS="-I$srcdir/opal/libltdl/"
# Must specifically mention $srcdir here for VPATH builds
# (this file is in the src tree).
AC_EGREP_HEADER([lt_dladvise_init], [*$srcdir*/opal/libltdl/ltdl.h],
[OPAL_HAVE_LTDL_ADVISE=1])

This was thesource of my mistake and confusion. In ompi we check for
"opal/libltdl/ltdl.h" and we do need -I$srcdir and in ompi-release we check
for "*$srcdir*/opal/libltdl/ltdl.h". I didn't noticed that wen did the
backport from ompi-release to ompi. I really thought that this files are
equal.

I think we need to converge to the unified solution.


2014-12-03 10:23 GMT+06:00 Ralph Castain :

> It is working for me, but I’m not sure if that is because of these changes
> or if it always worked for me. I haven’t tested the slurm integration in
> awhile.
>
>
> On Dec 2, 2014, at 7:59 PM, Artem Polyakov  wrote:
>
> Howard, does current mater fix your problems?
>
> среда, 3 декабря 2014 г. пользователь Artem Polyakov написал:
>
>>
>> 2014-12-03 8:30 GMT+06:00 Jeff Squyres (jsquyres) :
>>
>>> On Dec 2, 2014, at 8:43 PM, Artem Polyakov  wrote:
>>>
>>> > Jeff, your fix brakes my system again. Actually you just reverted my
>>> changes.
>>>
>>> No, I didn't just revert them -- I made changes.  I did forget about the
>>> second -I, though (to be fair, the 2nd -I was the *only* -I in there before
>>> I committed).
>>>
>> Yeah! I was speaking figurally :).
>>
>>
>>> Sorry about that -- I've tested your change (without the trailing /) and
>>> it seems to work ok.  I'd go ahead and merge.
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/devel/2014/12/16414.php
>>>
>>
>>
>>
>> --
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
>>
>
>
> --
> -
> Best regards, Artem Polyakov
> (Mobile mail)
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16416.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16417.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

среда, 3 декабря 2014 г. пользователь Jeff Squyres (jsquyres) написал:

> They were equivalent until yesterday.  :-)

I see. Got that!

>
> I was going to file a PR to bring the changes over to v1.8, but not until
> they had shaken out on master.
>
> Would you mind filing a PR?

Sure, will do that asap.


>

>
> On Dec 3, 2014, at 5:56 AM, Artem Polyakov  > wrote:
>
> > I finally found the clear reason of this strange situation!
> >
> > In ompi opal_setup_libltdl.m4 has the following content:
> > CPPFLAGS="-I$srcdir -I$srcdir/opal/libltdl"
> > AC_EGREP_HEADER([lt_dladvise_init], [opal/libltdl/ltdl.h],
> > [OPAL_HAVE_LTDL_ADVISE=1])
> >
> > And in ompi-release opal_setup_libltdl.m4:
> > CPPFLAGS="-I$srcdir/opal/libltdl/"
> > # Must specifically mention $srcdir here for VPATH builds
> > # (this file is in the src tree).
> > AC_EGREP_HEADER([lt_dladvise_init], [$srcdir/opal/libltdl/ltdl.h],
> >   [OPAL_HAVE_LTDL_ADVISE=1])
> >
> > This was thesource of my mistake and confusion. In ompi we check for
> "opal/libltdl/ltdl.h" and we do need -I$srcdir and in ompi-release we check
> for "$srcdir/opal/libltdl/ltdl.h". I didn't noticed that wen did the
> backport from ompi-release to ompi. I really thought that this files are
> equal.
> >
> > I think we need to converge to the unified solution.
> >
> >
> > 2014-12-03 10:23 GMT+06:00 Ralph Castain  >:
> > It is working for me, but I’m not sure if that is because of these
> changes or if it always worked for me. I haven’t tested the slurm
> integration in awhile.
> >
> >
> >> On Dec 2, 2014, at 7:59 PM, Artem Polyakov  > wrote:
> >>
> >> Howard, does current mater fix your problems?
> >>
> >> среда, 3 декабря 2014 г. пользователь Artem Polyakov написал:
> >>
> >> 2014-12-03 8:30 GMT+06:00 Jeff Squyres (jsquyres)  >:
> >> On Dec 2, 2014, at 8:43 PM, Artem Polyakov  > wrote:
> >>
> >> > Jeff, your fix brakes my system again. Actually you just reverted my
> changes.
> >>
> >> No, I didn't just revert them -- I made changes.  I did forget about
> the second -I, though (to be fair, the 2nd -I was the *only* -I in there
> before I committed).
> >> Yeah! I was speaking figurally :).
> >>
> >> Sorry about that -- I've tested your change (without the trailing /)
> and it seems to work ok.  I'd go ahead and merge.
> >>
> >> --
> >> Jeff Squyres
> >> jsquy...@cisco.com 
> >> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >>
> >> ___
> >> devel mailing list
> >> de...@open-mpi.org 
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16414.php
> >>
> >>
> >>
> >> --
> >> С Уважением, Поляков Артем Юрьевич
> >> Best regards, Artem Y. Polyakov
> >>
> >>
> >> --
> >> -
> >> Best regards, Artem Polyakov
> >> (Mobile mail)
> >> ___
> >> devel mailing list
> >> de...@open-mpi.org 
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16416.php
> >
> >
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16417.php
> >
> >
> >
> > --
> > С Уважением, Поляков Артем Юрьевич
> > Best regards, Artem Y. Polyakov
> > ___
> > devel mailing list
> > de...@open-mpi.org 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16421.php
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com 
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16422.php



-- 
-
Best regards, Artem Polyakov
(Mobile mail)

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

Jeff, I must admit that I don't completely understand how your fix work.
Can you explan me why this veriant was failing:

CPPFLAGS="-I$srcdir/opal/libltdl/"
AC_EGREP_HEADER([lt_dladvise_init], [$srcdir/opal/libltdl/ltdl.h]

while the new one:

CPPFLAGS="-I$srcdir -I$srcdir/opal/libltdl/"
AC_EGREP_HEADER([lt_dladvise_init], [opal/libltdl/ltdl.h],
 [OPAL_HAVE_LTDL_ADVISE=1])

works well?

Is there additional header files that are included in conftest.c and has to
be reached through $srcdir?

2014-12-03 20:51 GMT+06:00 Jeff Squyres (jsquyres) :

> Thanks!
>
> On Dec 3, 2014, at 7:03 AM, Artem Polyakov  wrote:
>
> >
> >
> > среда, 3 декабря 2014 г. пользователь Jeff Squyres (jsquyres) написал:
> > They were equivalent until yesterday.  :-)
> > I see. Got that!
> >
> > I was going to file a PR to bring the changes over to v1.8, but not
> until they had shaken out on master.
> >
> > Would you mind filing a PR?
> > Sure, will do that asap.
> >
> >
> >
> >
> > On Dec 3, 2014, at 5:56 AM, Artem Polyakov  wrote:
> >
> > > I finally found the clear reason of this strange situation!
> > >
> > > In ompi opal_setup_libltdl.m4 has the following content:
> > > CPPFLAGS="-I$srcdir -I$srcdir/opal/libltdl"
> > > AC_EGREP_HEADER([lt_dladvise_init], [opal/libltdl/ltdl.h],
> > > [OPAL_HAVE_LTDL_ADVISE=1])
> > >
> > > And in ompi-release opal_setup_libltdl.m4:
> > > CPPFLAGS="-I$srcdir/opal/libltdl/"
> > > # Must specifically mention $srcdir here for VPATH builds
> > > # (this file is in the src tree).
> > > AC_EGREP_HEADER([lt_dladvise_init], [$srcdir/opal/libltdl/ltdl.h],
> > >   [OPAL_HAVE_LTDL_ADVISE=1])
> > >
> > > This was thesource of my mistake and confusion. In ompi we check for
> "opal/libltdl/ltdl.h" and we do need -I$srcdir and in ompi-release we check
> for "$srcdir/opal/libltdl/ltdl.h". I didn't noticed that wen did the
> backport from ompi-release to ompi. I really thought that this files are
> equal.
> > >
> > > I think we need to converge to the unified solution.
> > >
> > >
> > > 2014-12-03 10:23 GMT+06:00 Ralph Castain :
> > > It is working for me, but I’m not sure if that is because of these
> changes or if it always worked for me. I haven’t tested the slurm
> integration in awhile.
> > >
> > >
> > >> On Dec 2, 2014, at 7:59 PM, Artem Polyakov 
> wrote:
> > >>
> > >> Howard, does current mater fix your problems?
> > >>
> > >> среда, 3 декабря 2014 г. пользователь Artem Polyakov написал:
> > >>
> > >> 2014-12-03 8:30 GMT+06:00 Jeff Squyres (jsquyres)  >:
> > >> On Dec 2, 2014, at 8:43 PM, Artem Polyakov 
> wrote:
> > >>
> > >> > Jeff, your fix brakes my system again. Actually you just reverted
> my changes.
> > >>
> > >> No, I didn't just revert them -- I made changes.  I did forget about
> the second -I, though (to be fair, the 2nd -I was the *only* -I in there
> before I committed).
> > >> Yeah! I was speaking figurally :).
> > >>
> > >> Sorry about that -- I've tested your change (without the trailing /)
> and it seems to work ok.  I'd go ahead and merge.
> > >>
> > >> --
> > >> Jeff Squyres
> > >> jsquy...@cisco.com
> > >> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> > >>
> > >> ___
> > >> devel mailing list
> > >> de...@open-mpi.org
> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16414.php
> > >>
> > >>
> > >>
> > >> --
> > >> С Уважением, Поляков Артем Юрьевич
> > >> Best regards, Artem Y. Polyakov
> > >>
> > >>
> > >> --
> > >> -
> > >> Best regards, Artem Polyakov
> > >> (Mobile mail)
> > >> ___
> > >> devel mailing list
> > >> de...@open-mpi.org
> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16416.php
> > >
> > >
> > > ___

Re: [OMPI devel] RTLD_GLOBAL question

[Artem Polyakov]

2014-12-04 17:29 GMT+06:00 Jeff Squyres (jsquyres) :

> On Dec 3, 2014, at 11:35 PM, Artem Polyakov  wrote:
>
> > Jeff, I must admit that I don't completely understand how your fix work.
> Can you explan me why this veriant was failing:
> >
> > CPPFLAGS="-I$srcdir/opal/libltdl/"
> > AC_EGREP_HEADER([lt_dladvise_init], [$srcdir/opal/libltdl/ltdl.h]
> >
> > while the new one:
> >
> > CPPFLAGS="-I$srcdir -I$srcdir/opal/libltdl/"
> > AC_EGREP_HEADER([lt_dladvise_init], [opal/libltdl/ltdl.h],
> >  [OPAL_HAVE_LTDL_ADVISE=1])
> >
> > works well?
> >
> > Is there additional header files that are included in conftest.c and has
> to be reached through $srcdir?
>
> No, it was simpler than that: "." (i.e., $srcdir in a non-VPATH build) is
> not necessarily in the default include search path for <> files (which is
> what AC_EGREP_HEADER uses).  For example:
>
> -
> [3:24] savbu-usnic-a:~/g/ompi (topic/master-libfabric●)
> $ cat test.c
> #include <./opal/libltdl/ltdl.h>
> [3:24] savbu-usnic-a:~/g/ompi (topic/master-libfabric●)
> $ gcc -E test.c > /dev/null
> test.c:1:33: fatal error: ./opal/libltdl/ltdl.h: No such file or directory
>  #include <./opal/libltdl/ltdl.h>
>  ^
> compilation terminated.
> -
>
> Notice that if I don't have -I. (i.e., -I$srcdir), the above compilation
> fails because it can't find <./opal/libltdl/ltdl.h>.
>
> But if I add -I., then the file can be found:
>
> -
> [3:24] savbu-usnic-a:~/g/ompi (topic/master-libfabric●)
> $ gcc -E test.c -I. > /dev/null
> [3:25] savbu-usnic-a:~/g/ompi (topic/master-libfabric●)
> $ echo $status
> 0
> -
>
> And since we're -I$srcdir, there's no need to include $srcdir in the
> filename.  Indeed, if $srcdir==., then adding it in the filename is
> harmless.  But if $srcdir=/path/to/somewhere, it's actually a problem.
> Regardless, $srcdir should no longer be in the filename.
>
> The part I forgot was that your version of libtool also requires some sub
> header files in the $srcdir/opal/libltdl tree, so a -I for that also needs
> to be there.
>
> Make sense?
>
Yes. Thank you!


>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/12/16433.php




-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] Info about ORTE structure

[Artem Polyakov]

2015-03-26 17:58 GMT+06:00 Gianmario Pozzi :

> Hi everyone,
> I'm an italian M.Sc. student in Computer Engineering at Politecnico di
> Milano.
>
> My team and I are trying to integrate OpenMPI with a real time resource
> manager written by a group of students named BBQ (
> http://bosp.dei.polimi.it/ ). We are encountering some troubles, though.
>
> Our main issue is to understand how ORTE interacts with the resource
> manager, which parts of the code (if any) are executed on the "slave" nodes
> and which ones on the "master".
> We spent some time looking at the source code but we still have many
> doubts.
>

Hello,
check orte/mca/plm and orte/mca/ras
PLM - process lifecycle manager
RAS - resource allocation subsystem.

In RAS mpirun detects under which RM it works and gets the allocation.
in PLM spawn of remote processes is done.
mpirun spawns orted daemons on the slave nodes and all the rest is done
without RM intervention (IMHO).


>
> Thank you.
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2015/03/17157.php
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] Info about ORTE structure

[Artem Polyakov]

P.S. also check ESS (orte/mca/ess) for environment setup.

2015-03-26 18:06 GMT+06:00 Artem Polyakov :

>
> 2015-03-26 17:58 GMT+06:00 Gianmario Pozzi :
>
>> Hi everyone,
>> I'm an italian M.Sc. student in Computer Engineering at Politecnico di
>> Milano.
>>
>> My team and I are trying to integrate OpenMPI with a real time resource
>> manager written by a group of students named BBQ (
>> http://bosp.dei.polimi.it/ ). We are encountering some troubles, though.
>>
>> Our main issue is to understand how ORTE interacts with the resource
>> manager, which parts of the code (if any) are executed on the "slave" nodes
>> and which ones on the "master".
>> We spent some time looking at the source code but we still have many
>> doubts.
>>
>
> Hello,
> check orte/mca/plm and orte/mca/ras
> PLM - process lifecycle manager
> RAS - resource allocation subsystem.
>
> In RAS mpirun detects under which RM it works and gets the allocation.
> in PLM spawn of remote processes is done.
> mpirun spawns orted daemons on the slave nodes and all the rest is done
> without RM intervention (IMHO).
>
>
>>
>> Thank you.
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2015/03/17157.php
>>
>
>
>
> --
> С Уважением, Поляков Артем Юрьевич
> Best regards, Artem Y. Polyakov
>



-- 
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov

Re: [OMPI devel] PMIX deadlock

[Artem Polyakov]

Hello, is there any progress on this topic? This affects our PMIx
measurements.

2015-10-30 21:21 GMT+06:00 Ralph Castain :

> I’ve verified that the orte/util/listener thread is not being started, so
> I don’t think it should be involved in this problem.
>
> HTH
> Ralph
>
> On Oct 30, 2015, at 8:07 AM, Ralph Castain  wrote:
>
> Hmmm…there is a hook that would allow the PMIx server to utilize that
> listener thread, but we aren’t currently using it. Each daemon plus mpirun
> will call orte_start_listener, but nothing is currently registering and so
> the listener in that code is supposed to just return without starting the
> thread.
>
> So the only listener thread that should exist is the one inside the PMIx
> server itself. If something else is happening, then that would be a bug. I
> can look at the orte listener code to ensure that the thread isn’t
> incorrectly starting.
>
>
> On Oct 29, 2015, at 10:03 PM, George Bosilca  wrote:
>
> Some progress, that puzzles me but might help you understand. Once the
> deadlock appears, if I manually kill the MPI process on the node where the
> deadlock was created, the local orte daemon doesn't notice and will just
> keep waiting.
>
> Quick question: I am under the impression that the issue is not in the
> PMIX server but somewhere around the listener_thread_fn in
> orte/util/listener.c. Possible ?
>
>   George.
>
>
> On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain  wrote:
>
>> Should have also clarified: the prior fixes are indeed in the current
>> master.
>>
>> On Oct 28, 2015, at 12:42 AM, Ralph Castain  wrote:
>>
>> Nope - I was wrong. The correction on the client side consisted of
>> attempting to timeout if the blocking recv failed. We then modified the
>> blocking send/recv so they would handle errors.
>>
>> So that problem occurred -after- the server had correctly called accept.
>> The listener code is in
>> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c
>>
>> It looks to me like the only way we could drop the accept (assuming the
>> OS doesn’t lose it) is if the file descriptor lies outside the expected
>> range once we fall out of select:
>>
>>
>> /* Spin accepting connections until all active listen sockets
>>  * do not have any incoming connections, pushing each connection
>>  * onto the event queue for processing
>>  */
>> do {
>> accepted_connections = 0;
>> /* according to the man pages, select replaces the given
>> descriptor
>>  * set with a subset consisting of those descriptors that are
>> ready
>>  * for the specified operation - in this case, a read. So we
>> need to
>>  * first check to see if this file descriptor is included in
>> the
>>  * returned subset
>>  */
>> if (0 == FD_ISSET(pmix_server_globals.listen_socket,
>> &readfds)) {
>> /* this descriptor is not included */
>> continue;
>> }
>>
>> /* this descriptor is ready to be read, which means a
>> connection
>>  * request has been received - so harvest it. All we want to
>> do
>>  * here is accept the connection and push the info onto the
>> event
>>  * library for subsequent processing - we don't want to
>> actually
>>  * process the connection here as it takes too long, and so
>> the
>>  * OS might start rejecting connections due to timeout.
>>  */
>> pending_connection = PMIX_NEW(pmix_pending_connection_t);
>> event_assign(&pending_connection->ev, pmix_globals.evbase, -1,
>>  EV_WRITE, connection_handler,
>> pending_connection);
>> pending_connection->sd =
>> accept(pmix_server_globals.listen_socket,
>> (struct
>> sockaddr*)&(pending_connection->addr),
>> &addrlen);
>> if (pending_connection->sd < 0) {
>> PMIX_RELEASE(pending_connection);
>> if (pmix_socket_errno != EAGAIN ||
>> pmix_socket_errno != EWOULDBLOCK) {
>> if (EMFILE == pmix_socket_errno) {
>> PMIX_ERROR_LOG(PMIX_ERR_OUT_OF_RESOURCE);
>> } else {
>> pmix_output(0, "listen_thread: accept() failed:
>> %s (%d).",
>> strerror(pmix_socket_errno),
>> pmix_socket_errno);
>> }
>> goto done;
>> }
>> continue;
>> }
>>
>> pmix_output_verbose(8, pmix_globals.debug_output,
>> "listen_thread: new connection: (%d, %d)",
>> pending_connection->sd,
>> pmix_socket_errno);
>> /* activate the event */
>> event_active(&pending_connection->ev, EV_WRITE, 1);
>>

Re: [OMPI devel] PMIX deadlock

[Artem Polyakov]

 you can try
>>>> to talk to a Unix Domain Socket too quickly - i.e., you open it and then
>>>> send to it, but the OS hasn’t yet set it up. In those cases, you can hang
>>>> the socket. However, I’ve tried adding some artificial delay, and while it
>>>> helped, it didn’t completely solve the problem.
>>>>
>>>> I have an idea for a workaround (set a timer and retry after awhile),
>>>> but would obviously prefer a real solution. I’m not even sure it will work
>>>> as it is unclear that the server (who is the one hung in accept) will break
>>>> free if the client closes the socket and retries.
>>>>
>>>>
>>>> On Nov 6, 2015, at 10:53 PM, Artem Polyakov  wrote:
>>>>
>>>> Hello, is there any progress on this topic? This affects our PMIx
>>>> measurements.
>>>>
>>>> 2015-10-30 21:21 GMT+06:00 Ralph Castain :
>>>>
>>>>> I’ve verified that the orte/util/listener thread is not being started,
>>>>> so I don’t think it should be involved in this problem.
>>>>>
>>>>> HTH
>>>>> Ralph
>>>>>
>>>>> On Oct 30, 2015, at 8:07 AM, Ralph Castain  wrote:
>>>>>
>>>>> Hmmm…there is a hook that would allow the PMIx server to utilize that
>>>>> listener thread, but we aren’t currently using it. Each daemon plus mpirun
>>>>> will call orte_start_listener, but nothing is currently registering and so
>>>>> the listener in that code is supposed to just return without starting the
>>>>> thread.
>>>>>
>>>>> So the only listener thread that should exist is the one inside the
>>>>> PMIx server itself. If something else is happening, then that would be a
>>>>> bug. I can look at the orte listener code to ensure that the thread isn’t
>>>>> incorrectly starting.
>>>>>
>>>>>
>>>>> On Oct 29, 2015, at 10:03 PM, George Bosilca 
>>>>> wrote:
>>>>>
>>>>> Some progress, that puzzles me but might help you understand. Once the
>>>>> deadlock appears, if I manually kill the MPI process on the node where the
>>>>> deadlock was created, the local orte daemon doesn't notice and will just
>>>>> keep waiting.
>>>>>
>>>>> Quick question: I am under the impression that the issue is not in the
>>>>> PMIX server but somewhere around the listener_thread_fn in
>>>>> orte/util/listener.c. Possible ?
>>>>>
>>>>>   George.
>>>>>
>>>>>
>>>>> On Wed, Oct 28, 2015 at 3:56 AM, Ralph Castain 
>>>>> wrote:
>>>>>
>>>>>> Should have also clarified: the prior fixes are indeed in the current
>>>>>> master.
>>>>>>
>>>>>> On Oct 28, 2015, at 12:42 AM, Ralph Castain  wrote:
>>>>>>
>>>>>> Nope - I was wrong. The correction on the client side consisted of
>>>>>> attempting to timeout if the blocking recv failed. We then modified the
>>>>>> blocking send/recv so they would handle errors.
>>>>>>
>>>>>> So that problem occurred -after- the server had correctly called
>>>>>> accept. The listener code is in
>>>>>> opal/mca/pmix/pmix1xx/pmix/src/server/pmix_server_listener.c
>>>>>>
>>>>>> It looks to me like the only way we could drop the accept (assuming
>>>>>> the OS doesn’t lose it) is if the file descriptor lies outside the 
>>>>>> expected
>>>>>> range once we fall out of select:
>>>>>>
>>>>>>
>>>>>> /* Spin accepting connections until all active listen sockets
>>>>>>  * do not have any incoming connections, pushing each
>>>>>> connection
>>>>>>  * onto the event queue for processing
>>>>>>  */
>>>>>> do {
>>>>>> accepted_connections = 0;
>>>>>> /* according to the man pages, select replaces the given
>>>>>> descriptor
>>>>>>  * set with a subset consisting of those descriptors that
>>>>>> are ready
>>>>>>  * for the specified operation - in this case, a read. 

Re: [OMPI devel] PMIX deadlock

[Artem Polyakov]

2015-11-09 22:42 GMT+06:00 Artem Polyakov :

> This is the very good point, Nysal!
>
> This is definitely a problem and I can say even more: avg. 3 from every 10
> tasks was affected by this bug. Once the PR (
> https://github.com/pmix/master/pull/8) was applied I was able to run 100
> testing tasks without any hangs.
>
> Here some more information on my symptoms. I was observing this without
> OMPI, just running pmix_client test binary from PMIx test suite with SLURM
> PMIx plugin.
> Periodicaly application was hanging. Investigation shows that not all
> processes are able to initialize correctly.
> Here is how such client's backtrace looks like:
>

P.S. I think that this backtrace may be relevant to George's problem as
well. In my case not all of the processes was hanging in the
connect_to_server, most of them were able to move forward and reach Fence.
George, the backtrace that you've posted was the same on both processes or
it was the "random" one from one of them?


> (gdb) bt
> #0  0x7f1448f1b7eb in recv () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x7f144914c191 in pmix_usock_recv_blocking (sd=9,
> data=0x7fff367f7c64 "", size=4) at src/usock/usock.c:166
> #2  0x7f1449152d18 in recv_connect_ack (sd=9) at
> src/client/pmix_client.c:837
> #3  0x7f14491546bf in usock_connect (addr=0x7fff367f7d60) at
> src/client/pmix_client.c:1103
> #4  0x7f144914f94c in connect_to_server (address=0x7fff367f7d60,
> cbdata=0x7fff367f7dd0) at src/client/pmix_client.c:179
> #5  0x7f1449150421 in PMIx_Init (proc=0x7fff367f81d0) at
> src/client/pmix_client.c:355
> #6  0x00401b97 in main (argc=9, argv=0x7fff367f83d8) at
> pmix_client.c:62
>
>
> The server-side debug has the following lines at the end of the file:
> [cn33:00482] pmix:server register client slurm.pmix.22.0:10
> [cn33:00482] pmix:server _register_client for nspace slurm.pmix.22.0 rank
> 10
> [cn33:00482] pmix:server setup_fork for nspace slurm.pmix.22.0 rank 10
>
> in normal operation the following lines should appear after lines above:
> 
> [cn33:00188] listen_thread: new connection: (26, 0)
> [cn33:00188] connection_handler: new connection: 26
> [cn33:00188] RECV CONNECT ACK FROM PEER ON SOCKET 26
> [cn33:00188] waiting for blocking recv of 16 bytes
> [cn33:00188] blocking receive complete from remote
> 
>
> At the client side I see the following lines
> cn33:00491] usock_peer_try_connect: attempting to connect to server
> [cn33:00491] usock_peer_try_connect: attempting to connect to server on
> socket 10
> [cn33:00491] pmix: SEND CONNECT ACK
> [cn33:00491] sec: native create_cred
> [cn33:00491] sec: using credential 1000:1000
> [cn33:00491] send blocking of 54 bytes to socket 10
> [cn33:00491] blocking send complete to socket 10
> [cn33:00491] pmix: RECV CONNECT ACK FROM SERVER
> [cn33:00491] waiting for blocking recv of 4 bytes
> [cn33:00491] blocking_recv received error 11:Resource temporarily
> unavailable from remote - cycling
> [cn33:00491] blocking_recv received error 11:Resource temporarily
> unavailable from remote - cycling
> [... repeated many times ...]
>
> With the fix for the problem highlighted by Nysal all runs cleanly.
>
>
> 2015-11-09 10:53 GMT+06:00 Nysal Jan K A :
>
>> In listen_thread():
>> 194 while (pmix_server_globals.listen_thread_active) {
>> 195 FD_ZERO(&readfds);
>> 196 FD_SET(pmix_server_globals.listen_socket, &readfds);
>> 197 max = pmix_server_globals.listen_socket;
>>
>> Is it possible that pmix_server_globals.listen_thread_active can be
>> false, in which case the thread just exits and will never call accept() ?
>>
>> In pmix_start_listening():
>> 147 /* fork off the listener thread */
>> 148 if (0 > pthread_create(&engine, NULL, listen_thread, NULL)) {
>> 149 return PMIX_ERROR;
>> 150 }
>> 151 pmix_server_globals.listen_thread_active = true;
>>
>> pmix_server_globals.listen_thread_active is set to true after the thread
>> is created, could this cause a race ?
>> listen_thread_active might also need to be declared as volatile.
>>
>> Regards
>> --Nysal
>>
>> On Sun, Nov 8, 2015 at 10:38 PM, George Bosilca 
>> wrote:
>>
>>> We had a power outage last week and the local disks on our cluster were
>>> wiped out. My tester was in there. But, I can rewrite it after SC.
>>>
>>>   George.
>>>
>>> On Sat, Nov 7, 2015 at 12:04 PM, Ralph Castain  wrote:
>>>
>>>> Could you send me your stress test? I’m wondering if it is just
>>>