Re: [OMPI users] slurm configuration override mpirun command line process mapping
Gilles, Adding ess component that excludes slurm and slurmd. I run into trouble about connection issue. I guess I need slurm and slurmd in my runtime context ! Anyway, as you mentioned that not a good solution regarding remaining mpi process when using scancel and I guess i will also lose some process monitoring functionalities from slurm. I will stick with mpirun command line update using --bynode option. Thanks a lot for your help. Regards, Nicolas 2018-05-17 14:23 GMT+02:00 Gilles Gouaillardet < gilles.gouaillar...@gmail.com>: > Nicolas, > > This looks odd at first glance, but as stated before, 1.6 is an obsolete > series. > A workaround could be to > mpirun—mca ess ... > And replace ... with a comma separated list of ess components that > excludes both slurm and slurmd. > > An other workaround could be to remove SLURM related environment variables > before calling mpirun. > > > Cheers, > > Gilles > > > On Thursday, May 17, 2018, Nicolas Deladerriere < > nicolas.deladerri...@gmail.com> wrote: > >> Hi all, >> >> Thanks for your feedback, >> >> about using " mpirun --mca ras ^slurm --mca plm ^slurm --mca ess >> ^slurm,slurmd ...". I am a bit confused since syntax sounds good, but I >> keep getting following error at run time : >> >> >> >> >> >> >> >> >> >> >> >> >> >> *--MCA >> framework parameters can only take a single negation operator("^"), and it >> must be at the beginning of the value. The followingvalue violates this >> rule:env,^slurm,slurmdWhen used, the negation operator sets the >> "exclusive" behavior mode,meaning that it will exclude all specified >> components (and implicitlyinclude all others). ..You cannot mix >> inclusive and exclusive behavior.* >> >> Is there other mca setting that could violate command line setting ? >> >> here is full mpirun command line : >> >> */../openmpi/1.6.5/bin/mpirun -prefix /.../openmpi/1.6.5 -tag-output -H >> r01n05 -x OMP_NUM_THREADS -np 1 --mca ras ^slurm --mca plm ^slurm --mca ess >> ^slurm,slurmd master_exe.x: -H r01n06,r01n07 -x OMP_NUM_THREADS -np 2 >> slave_exe.x* >> >> and ompi default setting : >> >> >> >> >> >> >> >> >> >> >> >> *host% ompi_info --all | grep slurm MCA ras: slurm (MCA >> v2.0, API v2.0, Component v1.6.5) MCA plm: slurm (MCA v2.0, >> API v2.0, Component v1.6.5) MCA ess: slurm (MCA v2.0, API >> v2.0, Component v1.6.5) MCA ess: slurmd (MCA v2.0, API >> v2.0, Component v1.6.5) MCA ras: parameter >> "ras_slurm_priority" (current value: <75>, data source: default >> value) Priority of the slurm ras >> component MCA plm: parameter "plm_slurm_args" (current >> value: , data source: default value) MCA plm: >> parameter "plm_slurm_priority" (current value: <0>, data source: default >> value) MCA ess: parameter "ess_slurm_priority" (current >> value: <0>, data source: default value) MCA ess: parameter >> "ess_slurmd_priority" (current value: <0>, data source: default value)* >> >> >> About "-H" option and using --bynode option: >> >> In my case, I do not specify number of slots by node to openmpi (see >> mpirun command just above). From what I see the only place I define number >> of slots in this case is actually through SLURM configuration >> (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken >> when running mpi processes. >> >> Using --bynode is probably the easiest solution in my case, even if I am >> scared that it will not necessary fit all my running configuration. Better >> solution would be to review my management script for better integration >> with slurm resources manager, but this is another story. >> >> Thanks for your help. >> Regards, >> Nicolas >> >> >> 2018-05-16 9:47 GMT+02:00 r...@open-mpi.org : >> >>> The problem here is that you have made an incorrect assumption. In the >>> older OMPI versions, the -H option simply indicated that the specified >>> hosts were available for use - it did not imply the number of slots on that >>> host. Since you have specified 2 slots on each host, and you told mpirun to >>> launch 2 procs of your second app_context (the “slave”), it filled the >>> first node with the 2 procs. >>> >>> I don’t recall the options for that old a version, but IIRC you should >>> add --pernode to the cmd line to get exactly 1 proc/node >>> >>> Or upgrade to a more recent OMPI version where -H can also be used to >>> specify the #slots on a node :-) >>> >>> >>> > On May 15, 2018, at 11:58 PM, Gilles Gouaillardet < >>> gilles.gouaillar...@gmail.com> wrote: >>> > >>> > You can try to disable SLURM : >>> > >>> > mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ... >>> > >>> > That will require you are able to SSH between compute nodes. >>> > Keep in mind this is far form ideal since it might leave some MPI >>> > processes on no
Re: [OMPI users] slurm configuration override mpirun command line process mapping
"mpirun takes the #slots for each node from the slurm allocation." Yes this is my issue and what I was not expected. But I will stick with --bynode solution. Thanks a lot for your help. Regards, Nicolas 2018-05-17 14:33 GMT+02:00 r...@open-mpi.org : > mpirun takes the #slots for each node from the slurm allocation. Your > hostfile (at least, what you provided) retained that information and shows > 2 slots on each node. So both the original allocation _and_ your > constructed hostfile are both telling mpirun to assign 2 slots on each node. > > Like I said before, on this old version, -H doesn’t say anything about > #slots - that information is coming solely from the original allocation and > your hostfile. > > > On May 17, 2018, at 5:11 AM, Nicolas Deladerriere < > nicolas.deladerri...@gmail.com> wrote: > > About "-H" option and using --bynode option: > > In my case, I do not specify number of slots by node to openmpi (see > mpirun command just above). From what I see the only place I define number > of slots in this case is actually through SLURM configuration > (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken > when running mpi processes. > > Using --bynode is probably the easiest solution in my case, even if I am > scared that it will not necessary fit all my running configuration. Better > solution would be to review my management script for better integration > with slurm resources manager, but this is another story. > > > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] slurm configuration override mpirun command line process mapping
mpirun takes the #slots for each node from the slurm allocation. Your hostfile (at least, what you provided) retained that information and shows 2 slots on each node. So both the original allocation _and_ your constructed hostfile are both telling mpirun to assign 2 slots on each node. Like I said before, on this old version, -H doesn’t say anything about #slots - that information is coming solely from the original allocation and your hostfile. > On May 17, 2018, at 5:11 AM, Nicolas Deladerriere > wrote: > > About "-H" option and using --bynode option: > > In my case, I do not specify number of slots by node to openmpi (see mpirun > command just above). From what I see the only place I define number of slots > in this case is actually through SLURM configuration > (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken when > running mpi processes. > > Using --bynode is probably the easiest solution in my case, even if I am > scared that it will not necessary fit all my running configuration. Better > solution would be to review my management script for better integration with > slurm resources manager, but this is another story. ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] slurm configuration override mpirun command line process mapping
Nicolas, This looks odd at first glance, but as stated before, 1.6 is an obsolete series. A workaround could be to mpirun—mca ess ... And replace ... with a comma separated list of ess components that excludes both slurm and slurmd. An other workaround could be to remove SLURM related environment variables before calling mpirun. Cheers, Gilles On Thursday, May 17, 2018, Nicolas Deladerriere < nicolas.deladerri...@gmail.com> wrote: > Hi all, > > Thanks for your feedback, > > about using " mpirun --mca ras ^slurm --mca plm ^slurm --mca ess > ^slurm,slurmd ...". I am a bit confused since syntax sounds good, but I > keep getting following error at run time : > > > > > > > > > > > > > > *--MCA > framework parameters can only take a single negation operator("^"), and it > must be at the beginning of the value. The followingvalue violates this > rule:env,^slurm,slurmdWhen used, the negation operator sets the > "exclusive" behavior mode,meaning that it will exclude all specified > components (and implicitlyinclude all others). ..You cannot mix > inclusive and exclusive behavior.* > > Is there other mca setting that could violate command line setting ? > > here is full mpirun command line : > > */../openmpi/1.6.5/bin/mpirun -prefix /.../openmpi/1.6.5 -tag-output -H > r01n05 -x OMP_NUM_THREADS -np 1 --mca ras ^slurm --mca plm ^slurm --mca ess > ^slurm,slurmd master_exe.x: -H r01n06,r01n07 -x OMP_NUM_THREADS -np 2 > slave_exe.x* > > and ompi default setting : > > > > > > > > > > > > *host% ompi_info --all | grep slurm MCA ras: slurm (MCA > v2.0, API v2.0, Component v1.6.5) MCA plm: slurm (MCA v2.0, > API v2.0, Component v1.6.5) MCA ess: slurm (MCA v2.0, API > v2.0, Component v1.6.5) MCA ess: slurmd (MCA v2.0, API > v2.0, Component v1.6.5) MCA ras: parameter > "ras_slurm_priority" (current value: <75>, data source: default > value) Priority of the slurm ras > component MCA plm: parameter "plm_slurm_args" (current > value: , data source: default value) MCA plm: > parameter "plm_slurm_priority" (current value: <0>, data source: default > value) MCA ess: parameter "ess_slurm_priority" (current > value: <0>, data source: default value) MCA ess: parameter > "ess_slurmd_priority" (current value: <0>, data source: default value)* > > > About "-H" option and using --bynode option: > > In my case, I do not specify number of slots by node to openmpi (see > mpirun command just above). From what I see the only place I define number > of slots in this case is actually through SLURM configuration > (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken > when running mpi processes. > > Using --bynode is probably the easiest solution in my case, even if I am > scared that it will not necessary fit all my running configuration. Better > solution would be to review my management script for better integration > with slurm resources manager, but this is another story. > > Thanks for your help. > Regards, > Nicolas > > > 2018-05-16 9:47 GMT+02:00 r...@open-mpi.org : > >> The problem here is that you have made an incorrect assumption. In the >> older OMPI versions, the -H option simply indicated that the specified >> hosts were available for use - it did not imply the number of slots on that >> host. Since you have specified 2 slots on each host, and you told mpirun to >> launch 2 procs of your second app_context (the “slave”), it filled the >> first node with the 2 procs. >> >> I don’t recall the options for that old a version, but IIRC you should >> add --pernode to the cmd line to get exactly 1 proc/node >> >> Or upgrade to a more recent OMPI version where -H can also be used to >> specify the #slots on a node :-) >> >> >> > On May 15, 2018, at 11:58 PM, Gilles Gouaillardet < >> gilles.gouaillar...@gmail.com> wrote: >> > >> > You can try to disable SLURM : >> > >> > mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ... >> > >> > That will require you are able to SSH between compute nodes. >> > Keep in mind this is far form ideal since it might leave some MPI >> > processes on nodes if you cancel a job, and mess SLURM accounting too. >> > >> > >> > Cheers, >> > >> > Gilles >> > >> > On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere >> > wrote: >> >> Hi all, >> >> >> >> >> >> >> >> I am trying to run mpi application through SLURM job scheduler. Here >> is my >> >> running sequence >> >> >> >> >> >> sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun >> >> >> >> >> >> In order to minimize modification of my production environment, I had >> to >> >> setup following hostlist management in different scripts: >> >> >> >> >> >> my_env_script.sh >> >> >> >> >> >> build host list from SLURM resource manager information >> >> >> >> Example: node01
Re: [OMPI users] slurm configuration override mpirun command line process mapping
Hi all, Thanks for your feedback, about using " mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ...". I am a bit confused since syntax sounds good, but I keep getting following error at run time : *--MCA framework parameters can only take a single negation operator("^"), and it must be at the beginning of the value. The followingvalue violates this rule:env,^slurm,slurmdWhen used, the negation operator sets the "exclusive" behavior mode,meaning that it will exclude all specified components (and implicitlyinclude all others). ..You cannot mix inclusive and exclusive behavior.* Is there other mca setting that could violate command line setting ? here is full mpirun command line : */../openmpi/1.6.5/bin/mpirun -prefix /.../openmpi/1.6.5 -tag-output -H r01n05 -x OMP_NUM_THREADS -np 1 --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd master_exe.x: -H r01n06,r01n07 -x OMP_NUM_THREADS -np 2 slave_exe.x* and ompi default setting : *host% ompi_info --all | grep slurm MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.5) MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.5) MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.5) MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.5) MCA ras: parameter "ras_slurm_priority" (current value: <75>, data source: default value) Priority of the slurm ras component MCA plm: parameter "plm_slurm_args" (current value: , data source: default value) MCA plm: parameter "plm_slurm_priority" (current value: <0>, data source: default value) MCA ess: parameter "ess_slurm_priority" (current value: <0>, data source: default value) MCA ess: parameter "ess_slurmd_priority" (current value: <0>, data source: default value)* About "-H" option and using --bynode option: In my case, I do not specify number of slots by node to openmpi (see mpirun command just above). From what I see the only place I define number of slots in this case is actually through SLURM configuration (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken when running mpi processes. Using --bynode is probably the easiest solution in my case, even if I am scared that it will not necessary fit all my running configuration. Better solution would be to review my management script for better integration with slurm resources manager, but this is another story. Thanks for your help. Regards, Nicolas 2018-05-16 9:47 GMT+02:00 r...@open-mpi.org : > The problem here is that you have made an incorrect assumption. In the > older OMPI versions, the -H option simply indicated that the specified > hosts were available for use - it did not imply the number of slots on that > host. Since you have specified 2 slots on each host, and you told mpirun to > launch 2 procs of your second app_context (the “slave”), it filled the > first node with the 2 procs. > > I don’t recall the options for that old a version, but IIRC you should add > --pernode to the cmd line to get exactly 1 proc/node > > Or upgrade to a more recent OMPI version where -H can also be used to > specify the #slots on a node :-) > > > > On May 15, 2018, at 11:58 PM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com> wrote: > > > > You can try to disable SLURM : > > > > mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ... > > > > That will require you are able to SSH between compute nodes. > > Keep in mind this is far form ideal since it might leave some MPI > > processes on nodes if you cancel a job, and mess SLURM accounting too. > > > > > > Cheers, > > > > Gilles > > > > On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere > > wrote: > >> Hi all, > >> > >> > >> > >> I am trying to run mpi application through SLURM job scheduler. Here is > my > >> running sequence > >> > >> > >> sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun > >> > >> > >> In order to minimize modification of my production environment, I had to > >> setup following hostlist management in different scripts: > >> > >> > >> my_env_script.sh > >> > >> > >> build host list from SLURM resource manager information > >> > >> Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2 > >> > >> > >> my_run_script.sh > >> > >> > >> Build host list according to required job (process mapping depends on > job > >> requirement). > >> > >> Nodes are always fully dedicated to my job, but I have to manage > different > >> master-slave situation with corresponding mpirun command: > >> > >> as many process as number of slots: > >> > >> mpirun -H node01 -np 1 process_master.x : -H > node02,node02,node03,node03 -np > >> 4 process_slave.x > >> > >> only one process per node (slots are usually used through openMP > threading) > >> > >> mpirun -H node01 -np 1 other_process
Re: [OMPI users] slurm configuration override mpirun command line process mapping
The problem here is that you have made an incorrect assumption. In the older OMPI versions, the -H option simply indicated that the specified hosts were available for use - it did not imply the number of slots on that host. Since you have specified 2 slots on each host, and you told mpirun to launch 2 procs of your second app_context (the “slave”), it filled the first node with the 2 procs. I don’t recall the options for that old a version, but IIRC you should add --pernode to the cmd line to get exactly 1 proc/node Or upgrade to a more recent OMPI version where -H can also be used to specify the #slots on a node :-) > On May 15, 2018, at 11:58 PM, Gilles Gouaillardet > wrote: > > You can try to disable SLURM : > > mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ... > > That will require you are able to SSH between compute nodes. > Keep in mind this is far form ideal since it might leave some MPI > processes on nodes if you cancel a job, and mess SLURM accounting too. > > > Cheers, > > Gilles > > On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere > wrote: >> Hi all, >> >> >> >> I am trying to run mpi application through SLURM job scheduler. Here is my >> running sequence >> >> >> sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun >> >> >> In order to minimize modification of my production environment, I had to >> setup following hostlist management in different scripts: >> >> >> my_env_script.sh >> >> >> build host list from SLURM resource manager information >> >> Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2 >> >> >> my_run_script.sh >> >> >> Build host list according to required job (process mapping depends on job >> requirement). >> >> Nodes are always fully dedicated to my job, but I have to manage different >> master-slave situation with corresponding mpirun command: >> >> as many process as number of slots: >> >> mpirun -H node01 -np 1 process_master.x : -H node02,node02,node03,node03 -np >> 4 process_slave.x >> >> only one process per node (slots are usually used through openMP threading) >> >> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 >> other_process_slave.x >> >> >> >> However, I realized that whatever I specified through my mpirun command, >> process mapping is overridden at run time by slurm according to slurm >> setting (either default setting or sbatch command line). For example, if I >> run with: >> >> >> sbatch -N 3 --exclusive my_env_script.sh myjob >> >> >> where final mpirun command (depending on myjob) is: >> >> >> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 >> other_process_slave.x >> >> >> It will be run with process mapping corresponding to: >> >> >> mpirun -H node01 -np 1 other_process_master.x : -H node02,node02 -np 2 >> other_process_slave.x >> >> >> So far I did not find a way to force mpirun to run with host mapping from >> command line instead of slurm one. Is there a way to do it (either by using >> MCA parameters of slurm configuration or …) ? >> >> >> openmpi version : 1.6.5 >> >> slurm version : 17.11.2 >> >> >> >> Ragards, >> >> Nicolas >> >> >> Note 1: I know, it would be better to let slurm manage my process mapping by >> only using slurm parameters and not specifying host mapping in my mpirun >> command, but in order to minimize modification in my production environment >> I had to use such solution. >> >> Note 2: I know I am using old openmpi version ! >> >> >> ___ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
Re: [OMPI users] slurm configuration override mpirun command line process mapping
You can try to disable SLURM : mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ... That will require you are able to SSH between compute nodes. Keep in mind this is far form ideal since it might leave some MPI processes on nodes if you cancel a job, and mess SLURM accounting too. Cheers, Gilles On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere wrote: > Hi all, > > > > I am trying to run mpi application through SLURM job scheduler. Here is my > running sequence > > > sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun > > > In order to minimize modification of my production environment, I had to > setup following hostlist management in different scripts: > > > my_env_script.sh > > > build host list from SLURM resource manager information > > Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2 > > > my_run_script.sh > > > Build host list according to required job (process mapping depends on job > requirement). > > Nodes are always fully dedicated to my job, but I have to manage different > master-slave situation with corresponding mpirun command: > > as many process as number of slots: > > mpirun -H node01 -np 1 process_master.x : -H node02,node02,node03,node03 -np > 4 process_slave.x > > only one process per node (slots are usually used through openMP threading) > > mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 > other_process_slave.x > > > > However, I realized that whatever I specified through my mpirun command, > process mapping is overridden at run time by slurm according to slurm > setting (either default setting or sbatch command line). For example, if I > run with: > > > sbatch -N 3 --exclusive my_env_script.sh myjob > > > where final mpirun command (depending on myjob) is: > > > mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 > other_process_slave.x > > > It will be run with process mapping corresponding to: > > > mpirun -H node01 -np 1 other_process_master.x : -H node02,node02 -np 2 > other_process_slave.x > > > So far I did not find a way to force mpirun to run with host mapping from > command line instead of slurm one. Is there a way to do it (either by using > MCA parameters of slurm configuration or …) ? > > > openmpi version : 1.6.5 > > slurm version : 17.11.2 > > > > Ragards, > > Nicolas > > > Note 1: I know, it would be better to let slurm manage my process mapping by > only using slurm parameters and not specifying host mapping in my mpirun > command, but in order to minimize modification in my production environment > I had to use such solution. > > Note 2: I know I am using old openmpi version ! > > > ___ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
[OMPI users] slurm configuration override mpirun command line process mapping
Hi all, I am trying to run mpi application through SLURM job scheduler. Here is my running sequence sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun In order to minimize modification of my production environment, I had to setup following hostlist management in different scripts: *my_env_script.sh* build host list from SLURM resource manager information Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2 *my_run_script.sh* Build host list according to required job (process mapping depends on job requirement). Nodes are always fully dedicated to my job, but I have to manage different master-slave situation with corresponding mpirun command: - as many process as number of slots: *mpirun -H node01 -np 1 process_master.x : -H node02,node02,node03,node03 -np 4 process_slave.x* - only one process per node (slots are usually used through openMP threading) *mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 other_process_slave.x* However, I realized that whatever I specified through my mpirun command, process mapping is overridden at run time by slurm according to slurm setting (either default setting or sbatch command line). For example, if I run with: *sbatch -N 3 --exclusive my_env_script.sh myjob* where final mpirun command (depending on myjob) is: *mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2 other_process_slave.x* It will be run with process mapping corresponding to: *mpirun -H node01 -np 1 other_process_master.x : -H node02,node02 -np 2 other_process_slave.x* So far I did not find a way to force mpirun to run with host mapping from command line instead of slurm one. Is there a way to do it (either by using MCA parameters of slurm configuration or …) ? openmpi version : 1.6.5 slurm version : 17.11.2 Ragards, Nicolas Note 1: I know, it would be better to let slurm manage my process mapping by only using slurm parameters and not specifying host mapping in my mpirun command, but in order to minimize modification in my production environment I had to use such solution. Note 2: I know I am using old openmpi version ! ___ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users