Hi, Gus and John,

my code (zeusmp2) is a F77 code ported in F95, and the very first time
I have launched it, the processed disappears almost immediately.
I checked the code with valgrind and some unallocated arrays were
passed wrongly to some subroutines.
After having corrected this bug, the code runs for a while and after
occour all the stuff described in my first post.
However, the code still performs a lot of main temporal cycle before
"die" (I don't know if thies piece of information is useful).

Now I'm going to check the memory usage, (I also have a lot of unused
variables in this pretty large code, maybe I shoud comment them).

uname -a returns
Linux cloud 2.6.9-42.0.3.ELsmp #1 SMP Thu Oct 5 16:29:37 CDT 2006
x86_64 x86_64 x86_64 GNU/Linux

ulimit -a returns:
core file size                        (blocks, -c) 0
data seg size                       (kbytes, -d) unlimited
file size                               (blocks, -f) unlimited
pending signals                    (-i) 1024
max locked memory             (kbytes, -l) 32
max memory size                (kbytes, -m) unlimited
open files                           (-n) 1024
pipe size                            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size                           (kbytes, -s) 10240
cpu time                             (seconds, -t) unlimited
max user processes             (-u) 36864
virtual memory                   (kbytes, -v) unlimited
file locks                            (-x) unlimited


I can log on the logins nodes, but unfortunately the command ls
/var/log/messages return
acpid               cron.4              messages.3         secure.4
anaconda.log        cups                messages.4         spooler
anaconda.syslog     dmesg               mpi_uninstall.log  spooler.1
anaconda.xlog       gdm                 ppp                spooler.2
audit               httpd               prelink.log        spooler.3
boot.log            itac_uninstall.log  rpmpkgs            spooler.4
boot.log.1          lastlog             rpmpkgs.1          vbox
boot.log.2          mail                rpmpkgs.2          wtmp
boot.log.3          maillog             rpmpkgs.3          wtmp.1
boot.log.4          maillog.1           rpmpkgs.4          Xorg.0.log
cmkl_install.log    maillog.2           samba              Xorg.0.log.old
cmkl_uninstall.log  maillog.3           scrollkeeper.log   yum.log
cron                maillog.4           secure             yum.log.1
cron.1              messages            secure.1
cron.2              messages.1          secure.2
cron.3              messages.2          secure.3

so, the log should be in some of these files (I don't have read
permission on these files). I'll contact the admin for that.

2012/9/1  <users-requ...@open-mpi.org>:
> Send users mailing list submissions to
>         us...@open-mpi.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://www.open-mpi.org/mailman/listinfo.cgi/users
> or, via email, send a message with subject or body 'help' to
>         users-requ...@open-mpi.org
>
> You can reach the person managing the list at
>         users-ow...@open-mpi.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of users digest..."
>
>
> Today's Topics:
>
>    1. Re: some mpi processes "disappear" on a cluster of        servers
>       (Gus Correa)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 31 Aug 2012 20:11:41 -0400
> From: Gus Correa <g...@ldeo.columbia.edu>
> Subject: Re: [OMPI users] some mpi processes "disappear" on a cluster
>         of      servers
> To: Open MPI Users <us...@open-mpi.org>
> Message-ID: <504152bd.3000...@ldeo.columbia.edu>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Andrea
>
> I would guess this is a memory problem.
> Do you know how much memory each node has?
> Do you know the memory that
> each MPI process in the CFD code requires?
> If the program starts swapping/paging into disk, because of
> low memory, those interesting things that you described can happen.
>
> I would login to the compute nodes and monitor the
> amount of memory being used with "top" right after the program
> starts to run.  If there is a pattern of which node tends to fail,
> track login to that fail-prone node and monitor it.
>
> Alternatively, if you cluster is running Ganglia,
> you can see the memory use graphically,
> in the Ganglia web page in a web browser.
>
> If your cluster
> doesn't allow direct user logins to compute nodes,
> you can ask the system administrator to do this for you.
>
> It may well be that the code has a memory leak, or that
> it has a memory request spike, which may not be caught by
> OpenMPI.
> [Jeff and Ralph will probably correct me soon for
> saying this, and I know the OpenMPI safeguards against
> process misbehavior are great, but ...]
>
> Anyway, we had several codes here that would make a node go south
> because of either type of memory problem, and subsequently the
> program would die, or the other processes in other nodes would
> continue "running" [i.e. mostly waiting for MPI calls to the
> dead node that would never return] as you described.
>
> If the problem is benign, i.e., if it is just that the
> memory-per-processor is not large enough to run in 10 processors,
> you can get around it by running in, say, 20 processors.
>
> Yet another issue that you may check is the stacksize in the
> compute nodes.  Many codes require a large stacksize, i.e.,
> they create large arrays in subroutines, etc, and
> the default stacksize in standard Linux distributions
> may not be as large as needed.
> We use ulimited stacksize in our compute nodes.
>
> You can ask the system administrator to check this for you,
> and perhaps change it in /etc/security/limits.conf to make it
> unlimited or at least larger than the default.
> The Linux shell command "ulimit -a" [bash] or
> "limit" [tcsh] will tell what the limits are.
>
> I hope this helps,
> Gus Correa
>
> On 08/31/2012 07:15 PM, Andrea Negri wrote:
>> Hi, I have been in trouble for a year.
>>
>> I run a pure MPI (no openMP) Fortran fluid dynamical code on a cluster
>> of server, and I obtain a strange behaviour by running the code on
>> multiple nodes.
>> The cluster is formed by 16 pc (1 pc is a node) with a dual core processor.
>> Basically, I'm able to run the code from the login node with the command:
>> mpirun  --mca btl_base_verbose 100 --mca backtrace_base_verbose 100
>> --mca memory_base_verbose 100 --mca sysinfo_base_verbose 100  -nolocal
>> -hostfile ./host_file -n 10  ./zeusmp2.x>>  zmp_errors 2>&1
>> by selecting one process per core (i.e. in this case I use 5 nodes)
>>
>> The code starts, and it runs correctely for some time.
>> After that, an entire node (sometimes two) "disappears" and it cannot
>> be reached with the ssh command, which returns: No route to host.
>> Sometimes the node is still reachable, but the two processes that was
>> running on the node are disappears.
>> In addition, on the other nodes, the others processes are still running.
>>
>> If I have a look on the output and error file of mpirun, the following
>> error is present: [btl_tcp_frag.c:215:mca_btl_tcp_frag_recv]
>> mca_btl_tcp_frag_recv: readv failed: No route to host (113)
>>
>> PS: I'm not the admin of the cluster, I've installed the gcc and
>> openmpi on my own because the complier aviable on that machine are 8
>> years old.
>>
>>
>> I post here some information, if you want other info, you have only to
>> tell me which command I have to type on the bash and I will
>> immediately reply.
>>
>>
>> complier: gcc 4.7 (which was also used to compile openmpi)
>> openmpi version: 1.6
>>
>> output of "ompi_info --all" from the node where I launch mpirun (which
>> is also the login node of the cluster)
>>
>>    Package: Open MPI and...@cloud.bo.astro.it Distribution
>>                  Open MPI: 1.6
>>     Open MPI SVN revision: r26429
>>     Open MPI release date: May 10, 2012
>>                  Open RTE: 1.6
>>     Open RTE SVN revision: r26429
>>     Open RTE release date: May 10, 2012
>>                      OPAL: 1.6
>>         OPAL SVN revision: r26429
>>         OPAL release date: May 10, 2012
>>                   MPI API: 2.1
>>              Ident string: 1.6
>>             MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6)
>>                MCA memory: linux (MCA v2.0, API v2.0, Component v1.6)
>>             MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA carto: file (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6)
>>             MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6)
>>             MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA timer: linux (MCA v2.0, API v2.0, Component v1.6)
>>           MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6)
>>           MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6)
>>               MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA hwloc: hwloc132 (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6)
>>                MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6)
>>             MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6)
>>             MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: basic (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: inter (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: self (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: sm (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: sync (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6)
>>                    MCA io: romio (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA pml: csum (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA pml: v (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6)
>>                MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA btl: self (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA btl: sm (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA topo: unity (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA iof: orted (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA iof: tool (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6)
>>                  MCA odls: default (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ras: cm (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA rml: oob (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: cm (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: direct (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: linear (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: radix (MCA v2.0, API v2.0, Component v1.6)
>>                MCA routed: slave (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6)
>>                 MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6)
>>                MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: env (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: slave (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6)
>>                   MCA ess: tool (MCA v2.0, API v2.0, Component v1.6)
>>               MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6)
>>               MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6)
>>               MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6)
>>              MCA notifier: command (MCA v2.0, API v1.0, Component v1.6)
>>              MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6)
>>                    Prefix: /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>               Exec_prefix: /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>                    Bindir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/bin
>>                   Sbindir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/sbin
>>                    Libdir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib
>>                    Incdir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/include
>>                    Mandir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/man
>>                 Pkglibdir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi
>>                Libexecdir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/libexec
>>               Datarootdir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share
>>                   Datadir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share
>>                Sysconfdir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc
>>            Sharedstatedir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/com
>>             Localstatedir: 
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/var
>>                   Infodir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/info
>>                Pkgdatadir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/openmpi
>>                 Pkglibdir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi
>>             Pkgincludedir:
>> /home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/include/openmpi
>>   Configured architecture: x86_64-unknown-linux-gnu
>>            Configure host: cloud.bo.astro.it
>>             Configured by: andrea
>>             Configured on: Tue Jul 31 10:53:46 CEST 2012
>>            Configure host: cloud.bo.astro.it
>>                  Built by: andrea
>>                  Built on: Tue Jul 31 11:08:33 CEST 2012
>>                Built host: cloud.bo.astro.it
>>                C bindings: yes
>>              C++ bindings: yes
>>        Fortran77 bindings: yes (all)
>>        Fortran90 bindings: yes
>>   Fortran90 bindings size: medium
>>                C compiler: /home/andrea/library/gcc/gcc-objects/bin/gcc
>>       C compiler absolute:
>>    C compiler family name: GNU
>>        C compiler version: 4.7.1
>>               C char size: 1
>>               C bool size: 1
>>              C short size: 2
>>                C int size: 4
>>               C long size: 8
>>              C float size: 4
>>             C double size: 8
>>            C pointer size: 8
>>              C char align: 1
>>              C bool align: 1
>>               C int align: 4
>>             C float align: 4
>>            C double align: 8
>>              C++ compiler: /home/andrea/library/gcc/gcc-objects/bin/g++
>>     C++ compiler absolute: none
>>        Fortran77 compiler: /home/andrea/library/gcc/gcc-objects/bin/gfortran
>>    Fortran77 compiler abs:
>>        Fortran90 compiler: /home/andrea/library/gcc/gcc-objects/bin/gfortran
>>    Fortran90 compiler abs:
>>         Fort integer size: 4
>>         Fort logical size: 4
>>   Fort logical value true: 1
>>        Fort have integer1: yes
>>        Fort have integer2: yes
>>        Fort have integer4: yes
>>        Fort have integer8: yes
>>       Fort have integer16: no
>>           Fort have real4: yes
>>           Fort have real8: yes
>>          Fort have real16: no
>>        Fort have complex8: yes
>>       Fort have complex16: yes
>>       Fort have complex32: no
>>        Fort integer1 size: 1
>>        Fort integer2 size: 2
>>        Fort integer4 size: 4
>>        Fort integer8 size: 8
>>       Fort integer16 size: -1
>>            Fort real size: 4
>>           Fort real4 size: 4
>>           Fort real8 size: 8
>>          Fort real16 size: 16
>>        Fort dbl prec size: 8
>>            Fort cplx size: 8
>>        Fort dbl cplx size: 16
>>           Fort cplx8 size: 8
>>          Fort cplx16 size: 16
>>          Fort cplx32 size: 32
>>        Fort integer align: 4
>>       Fort integer1 align: 1
>>       Fort integer2 align: 2
>>       Fort integer4 align: 4
>>       Fort integer8 align: 8
>>      Fort integer16 align: -1
>>           Fort real align: 4
>>          Fort real4 align: 4
>>          Fort real8 align: 8
>>         Fort real16 align: 16
>>       Fort dbl prec align: 8
>>           Fort cplx align: 4
>>       Fort dbl cplx align: 8
>>          Fort cplx8 align: 4
>>         Fort cplx16 align: 8
>>         Fort cplx32 align: 16
>>               C profiling: yes
>>             C++ profiling: yes
>>       Fortran77 profiling: yes
>>       Fortran90 profiling: yes
>>            C++ exceptions: no
>>            Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
>>             Sparse Groups: no
>>              Build CFLAGS: -DNDEBUG -g -O2 -finline-functions
>> -fno-strict-aliasing
>>                            -pthread
>>            Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
>>              Build FFLAGS:
>>             Build FCFLAGS:
>>             Build LDFLAGS: -Wl,--rpath
>> -Wl,/home/andrea/library/gcc/gcc-objects/lib64
>>                Build LIBS: -lrt -lnsl  -lutil -lm
>>      Wrapper extra CFLAGS: -pthread
>>    Wrapper extra CXXFLAGS: -pthread
>>      Wrapper extra FFLAGS: -pthread
>>     Wrapper extra FCFLAGS: -pthread
>>     Wrapper extra LDFLAGS:
>>        Wrapper extra LIBS: -ldl  -lm -lnuma -lrt -lnsl  -lutil -lm
>>    Internal debug support: no
>>    MPI interface warnings: yes
>>       MPI parameter check: runtime
>> Memory profiling support: no
>> Memory debugging support: no
>>           libltdl support: no
>>     Heterogeneous support: no
>>   mpirun default --prefix: yes
>>           MPI I/O support: yes
>>         MPI_WTIME support: gettimeofday
>>       Symbol vis. support: yes
>>     Host topology support: yes
>>            MPI extensions: affinity example
>>     FT Checkpoint support: no (checkpoint thread: no)
>>       VampirTrace support: yes
>>    MPI_MAX_PROCESSOR_NAME: 256
>>      MPI_MAX_ERROR_STRING: 256
>>       MPI_MAX_OBJECT_NAME: 64
>>          MPI_MAX_INFO_KEY: 36
>>          MPI_MAX_INFO_VAL: 256
>>         MPI_MAX_PORT_NAME: 1024
>>    MPI_MAX_DATAREP_STRING: 128
>>                   MCA mca: parameter "mca_param_files" (current value:
>>
>> </home/andrea/.openmpi/mca-params.conf:/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc/openmpi-mca-params.conf>,
>>                            data source: default value)
>>                            Path for MCA configuration files containing
>> default parameter
>>                            values
>>                   MCA mca: parameter "mca_base_param_file_prefix"
>> (current value:<none>,
>>                            data source: default value)
>>                            Aggregate MCA parameter file sets
>>                   MCA mca: parameter "mca_base_param_file_path" (current 
>> value:
>>
>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/share/openmpi/amca-param-sets:/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/bin>,
>>                            data source: default value)
>>                            Aggregate MCA parameter Search path
>>                   MCA mca: parameter "mca_base_param_file_path_force"
>> (current value:
>>                            <none>, data source: default value)
>>                            Forced Aggregate MCA parameter Search path
>>                   MCA mca: parameter "mca_component_path" (current value:
>>
>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/lib/openmpi:/home/andrea/.openmpi/components>,
>>                            data source: default value)
>>                            Path where to look for Open MPI and ORTE 
>> components
>>                   MCA mca: parameter "mca_component_show_load_errors"
>> (current value:<1>,
>>                            data source: default value)
>>                            Whether to show errors for components that
>> failed to load or
>>                            not
>>                   MCA mca: parameter "mca_component_disable_dlopen"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether to attempt to disable opening
>> dynamic components or not
>>                   MCA mca: parameter "mca_verbose" (current value:
>> <stderr>, data source:
>>                            default value)
>>                            Specifies where the default error output
>> stream goes (this is
>>                            separate from distinct help messages).  Accepts a
>>                            comma-delimited list of: stderr, stdout, syslog,
>>                            syslogpri:<notice|info|debug>,
>> syslogid:<str>  (where str is the
>>                            prefix string for all syslog notices),
>> file[:filename] (if
>>                            filename is not specified, a default
>> filename is used),
>>                            fileappend (if not specified, the file is opened 
>> for
>>                            truncation), level[:N] (if specified,
>> integer verbose level;
>>                            otherwise, 0 is implied)
>>                   MCA mpi: parameter "mpi_paffinity_alone" (current
>> value:<0>, data
>>                            source: default value, synonym of:
>> opal_paffinity_alone)
>>                            If nonzero, assume that this job is the only (set 
>> of)
>>                            process(es) running on each node and bind 
>> processes to
>>                            processors, starting with processor ID 0
>>                   MCA mpi: parameter "mpi_param_check" (current value:
>> <1>, data source:
>>                            default value)
>>                            Whether you want MPI API parameters checked
>> at run-time or not.
>>                             Possible values are 0 (no checking) and 1
>> (perform checking at
>>                            run-time)
>>                   MCA mpi: parameter "mpi_yield_when_idle" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Yield the processor when waiting for MPI
>> communication (for MPI
>>                            processes, will default to 1 when
>> oversubscribing nodes)
>>                   MCA mpi: parameter "mpi_event_tick_rate" (current
>> value:<-1>, data
>>                            source: default value)
>>                            How often to progress TCP communications (0
>> = never, otherwise
>>                            specified in microseconds)
>>                   MCA mpi: parameter "mpi_show_handle_leaks" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether MPI_FINALIZE shows all MPI handles
>> that were not freed
>>                            or not
>>                   MCA mpi: parameter "mpi_no_free_handles" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether to actually free MPI objects when
>> their handles are
>>                            freed
>>                   MCA mpi: parameter "mpi_show_mpi_alloc_mem_leaks"
>> (current value:<0>,
>>                            data source: default value)
>>                            If>0, MPI_FINALIZE will show up to this
>> many instances of
>>                            memory allocated by MPI_ALLOC_MEM that was
>> not freed by
>>                            MPI_FREE_MEM
>>                   MCA mpi: parameter "mpi_show_mca_params" (current
>> value:<none>, data
>>                            source: default value)
>>                            Whether to show all MCA parameter values
>> during MPI_INIT or not
>>                            (good for reproducability of MPI jobs for
>> debug purposes).
>>                            Accepted values are all, default, file, api,
>> and enviro - or a
>>                            comma delimited combination of them
>>                   MCA mpi: parameter "mpi_show_mca_params_file"
>> (current value:<none>,
>>                            data source: default value)
>>                            If mpi_show_mca_params is true, setting this
>> string to a valid
>>                            filename tells Open MPI to dump all the MCA
>> parameter values
>>                            into a file suitable for reading via the
>> mca_param_files
>>                            parameter (good for reproducability of MPI jobs)
>>                   MCA mpi: parameter "mpi_keep_peer_hostnames" (current
>> value:<1>, data
>>                            source: default value)
>>                            If nonzero, save the string hostnames of all
>> MPI peer processes
>>                            (mostly for error / debugging output
>> messages).  This can add
>>                            quite a bit of memory usage to each MPI process.
>>                   MCA mpi: parameter "mpi_abort_delay" (current value:
>> <0>, data source:
>>                            default value)
>>                            If nonzero, print out an identifying message
>> when MPI_ABORT is
>>                            invoked (hostname, PID of the process that
>> called MPI_ABORT)
>>                            and delay for that many seconds before
>> exiting (a negative
>>                            delay value means to never abort).  This
>> allows attaching of a
>>                            debugger before quitting the job.
>>                   MCA mpi: parameter "mpi_abort_print_stack" (current
>> value:<0>, data
>>                            source: default value)
>>                            If nonzero, print out a stack trace when
>> MPI_ABORT is invoked
>>                   MCA mpi: parameter "mpi_preconnect_mpi" (current
>> value:<0>, data
>>                            source: default value, synonyms: 
>> mpi_preconnect_all)
>>                            Whether to force MPI processes to fully
>> wire-up the MPI
>>                            connections between MPI processes during
>> MPI_INIT (vs. making
>>                            connections lazily -- upon the first MPI
>> traffic between each
>>                            process peer pair)
>>                   MCA mpi: parameter "mpi_preconnect_all" (current
>> value:<0>, data
>>                            source: default value, deprecated, synonym of:
>>                            mpi_preconnect_mpi)
>>                            Whether to force MPI processes to fully
>> wire-up the MPI
>>                            connections between MPI processes during
>> MPI_INIT (vs. making
>>                            connections lazily -- upon the first MPI
>> traffic between each
>>                            process peer pair)
>>                   MCA mpi: parameter "mpi_leave_pinned" (current value:
>> <-1>, data source:
>>                            default value)
>>                            Whether to use the "leave pinned" protocol
>> or not.  Enabling
>>                            this setting can help bandwidth performance
>> when repeatedly
>>                            sending and receiving large messages with
>> the same buffers over
>>                            RDMA-based networks (0 = do not use "leave
>> pinned" protocol, 1
>>                            = use "leave pinned" protocol, -1 = allow
>> network to choose at
>>                            runtime).
>>                   MCA mpi: parameter "mpi_leave_pinned_pipeline"
>> (current value:<0>, data
>>                            source: default value)
>>                            Whether to use the "leave pinned pipeline"
>> protocol or not.
>>                   MCA mpi: parameter "mpi_warn_on_fork" (current value:
>> <1>, data source:
>>                            default value)
>>                            If nonzero, issue a warning if program forks
>> under conditions
>>                            that could cause system errors
>>                   MCA mpi: information "mpi_have_sparse_group_storage"
>> (value:<0>, data
>>                            source: default value)
>>                            Whether this Open MPI installation supports
>> storing of data in
>>                            MPI groups in "sparse" formats (good for
>> extremely large
>>                            process count MPI jobs that create many
>> communicators/groups)
>>                   MCA mpi: parameter "mpi_use_sparse_group_storage"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether to use "sparse" storage formats for
>> MPI groups (only
>>                            relevant if mpi_have_sparse_group_storage is 1)
>>                   MCA mpi: parameter "mpi_notify_init_finalize"
>> (current value:<1>, data
>>                            source: default value)
>>                            If nonzero, send two notifications during
>> MPI_INIT: one near
>>                            when MPI_INIT starts, and another right
>> before MPI_INIT
>>                            finishes, and send 2 notifications during
>> MPI_FINALIZE: one
>>                            right when MPI_FINALIZE starts, and another near 
>> when
>>                            MPI_FINALIZE finishes.
>>                  MCA orte: parameter "orte_base_help_aggregate"
>> (current value:<1>, data
>>                            source: default value)
>>                            If orte_base_help_aggregate is true,
>> duplicate help messages
>>                            will be aggregated rather than displayed
>> individually.  This
>>                            can be helpful for parallel jobs that
>> experience multiple
>>                            identical failures; rather than print out
>> the same help/failure
>>                            message N times, display it once with a
>> count of how many
>>                            processes sent the same message.
>>                  MCA orte: parameter "orte_tmpdir_base" (current value:
>> <none>, data
>>                            source: default value)
>>                            Base of the session directory tree
>>                  MCA orte: parameter "orte_no_session_dirs" (current
>> value:<none>, data
>>                            source: default value)
>>                            Prohibited locations for session directories 
>> (multiple
>>                            locations separated by ',', default=NULL)
>>                  MCA orte: parameter "orte_send_profile" (current
>> value:<0>, data source:
>>                            default value)
>>                            Send profile info in launch message
>>                  MCA orte: parameter "orte_debug" (current value:<0>,
>> data source:
>>                            default value)
>>                            Top-level ORTE debug switch (default verbosity: 1)
>>                  MCA orte: parameter "orte_debug_verbose" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Verbosity level for ORTE debug messages (default: 
>> 1)
>>                  MCA orte: parameter "orte_debug_daemons" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether to debug the ORTE daemons or not
>>                  MCA orte: parameter "orte_debug_daemons_file" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether want stdout/stderr of daemons to go
>> to a file or not
>>                  MCA orte: parameter "orte_daemon_bootstrap" (current
>> value:<0>, data
>>                            source: default value)
>>                            Bootstrap the connection to the HNP
>>                  MCA orte: parameter "orte_leave_session_attached"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether applications and/or daemons should
>> leave their sessions
>>                            attached so that any output can be received
>> - this allows X
>>                            forwarding without all the attendant debugging 
>> output
>>                  MCA orte: parameter "orte_output_debugger_proctable"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether or not to output the debugger
>> proctable after launch
>>                            (default: false)
>>                  MCA orte: parameter "orte_debugger_test_daemon"
>> (current value:<none>,
>>                            data source: default value)
>>                            Name of the executable to be used to
>> simulate a debugger
>>                            colaunch (relative or absolute path)
>>                  MCA orte: parameter "orte_debugger_test_attach"
>> (current value:<0>, data
>>                            source: default value)
>>                            Test debugger colaunch after debugger attachment
>>                  MCA orte: parameter "orte_debugger_check_rate"
>> (current value:<0>, data
>>                            source: default value)
>>                            Set rate (in secs) for auto-detect of
>> debugger attachment (0 =>
>>                            do not check)
>>                  MCA orte: parameter "orte_do_not_launch" (current
>> value:<0>, data
>>                            source: default value)
>>                            Perform all necessary operations to prepare
>> to launch the
>>                            application, but do not actually launch it
>>                  MCA orte: parameter "orte_daemon_spin" (current value:
>> <0>, data source:
>>                            default value)
>>                            Have any orteds spin until we can connect a
>> debugger to them
>>                  MCA orte: parameter "orte_daemon_fail" (current value:
>> <-1>, data source:
>>                            default value)
>>                            Have the specified orted fail after init for
>> debugging purposes
>>                  MCA orte: parameter "orte_daemon_fail_delay" (current
>> value:<0>, data
>>                            source: default value)
>>                            Have the specified orted fail after
>> specified number of seconds
>>                            (default: 0 =>  no delay)
>>                  MCA orte: parameter "orte_heartbeat_rate" (current
>> value:<0>, data
>>                            source: default value)
>>                            Seconds between checks for daemon
>> state-of-health (default: 0
>>                            =>  do not check)
>>                  MCA orte: parameter "orte_startup_timeout" (current
>> value:<0>, data
>>                            source: default value)
>>                            Milliseconds/daemon to wait for startup
>> before declaring
>>                            failed_to_start (default: 0 =>  do not check)
>>                  MCA orte: parameter "orte_timing" (current value:<0>,
>> data source:
>>                            default value)
>>                            Request that critical timing loops be measured
>>                  MCA orte: parameter "orte_timing_details" (current
>> value:<0>, data
>>                            source: default value)
>>                            Request that detailed timing data by reported
>>                  MCA orte: parameter "orte_base_user_debugger" (current
>> value:<totalview
>>                            @mpirun@ -a @mpirun_args@ : ddt -n @np@
>> -start @executable@
>>                            @executable_argv@ @single_app@ : fxp @mpirun@ -a
>>                            @mpirun_args@>, data source: default value)
>>                            Sequence of user-level debuggers to search
>> for in orterun
>>                  MCA orte: parameter "orte_abort_timeout" (current
>> value:<1>, data
>>                            source: default value)
>>                            Max time to wait [in secs] before aborting
>> an ORTE operation
>>                            (default: 1sec)
>>                  MCA orte: parameter "orte_timeout_step" (current
>> value:<1000>, data
>>                            source: default value)
>>                            Time to wait [in usecs/proc] before aborting
>> an ORTE operation
>>                            (default: 1000 usec/proc)
>>                  MCA orte: parameter "orte_default_hostfile" (current value:
>>
>> </home/andrea/library/openmpi/openmpi-1.6-gnu-4.7/etc/openmpi-default-hostfile>,
>>                            data source: default value)
>>                            Name of the default hostfile (relative or
>> absolute path, "none"
>>                            to ignore environmental or default MCA param 
>> setting)
>>                  MCA orte: parameter "orte_rankfile" (current value:
>> <none>, data source:
>>                            default value, synonyms: rmaps_rank_file_path)
>>                            Name of the rankfile to be used for mapping
>> processes (relative
>>                            or absolute path)
>>                  MCA orte: parameter "orte_keep_fqdn_hostnames"
>> (current value:<0>, data
>>                            source: default value)
>>                            Whether or not to keep FQDN hostnames [default: 
>> no]
>>                  MCA orte: parameter "orte_use_regexp" (current value:
>> <0>, data source:
>>                            default value)
>>                            Whether or not to use regular expressions
>> for launch [default:
>>                            no]
>>                  MCA orte: parameter "orte_tag_output" (current value:
>> <0>, data source:
>>                            default value)
>>                            Tag all output with [job,rank] (default: false)
>>                  MCA orte: parameter "orte_xml_output" (current value:
>> <0>, data source:
>>                            default value)
>>                            Display all output in XML format (default: false)
>>                  MCA orte: parameter "orte_xml_file" (current value:
>> <none>, data source:
>>                            default value)
>>                            Provide all output in XML format to the specified 
>> file
>>                  MCA orte: parameter "orte_timestamp_output" (current
>> value:<0>, data
>>                            source: default value)
>>                            Timestamp all application process output
>> (default: false)
>>                  MCA orte: parameter "orte_output_filename" (current
>> value:<none>, data
>>                            source: default value)
>>                            Redirect output from application processes
>> into filename.rank
>>                            [default: NULL]
>>                  MCA orte: parameter "orte_show_resolved_nodenames"
>> (current value:<0>,
>>                            data source: default value)
>>                            Display any node names that are resolved to
>> a different name
>>                            (default: false)
>>                  MCA orte: parameter "orte_hetero_apps" (current value:
>> <0>, data source:
>>                            default value)
>>                            Indicates that multiple app_contexts are
>> being provided that
>>                            are a mix of 32/64 bit binaries (default: false)
>>                  MCA orte: parameter "orte_launch_agent" (current
>> value:<orted>, data
>>                            source: default value)
>>                            Command used to start processes on remote
>> nodes (default:
>>                            orted)
>>                  MCA orte: parameter "orte_allocation_required"
>> (current value:<0>, data
>>                            source: default value)
>>                            Whether or not an allocation by a resource
>> manager is required
>>                            [default: no]
>>                  MCA orte: parameter "orte_xterm" (current value:
>> <none>, data source:
>>                            default value)
>>                            Create a new xterm window and display output
>> from the specified
>>                            ranks there [default: none]
>>                  MCA orte: parameter "orte_forward_job_control"
>> (current value:<0>, data
>>                            source: default value)
>>                            Forward SIGTSTP (after converting to
>> SIGSTOP) and SIGCONT
>>                            signals to the application procs [default: no]
>>                  MCA orte: parameter "orte_rsh_agent" (current value:
>> <ssh : rsh>, data
>>                            source: default value, synonyms:
>> pls_rsh_agent, plm_rsh_agent)
>>                            The command used to launch executables on remote 
>> nodes
>>                            (typically either "ssh" or "rsh")
>>                  MCA orte: parameter "orte_assume_same_shell" (current
>> value:<1>, data
>>                            source: default value, synonyms:
>> plm_rsh_assume_same_shell)
>>                            If set to 1, assume that the shell on the
>> remote node is the
>>                            same as the shell on the local node.
>> Otherwise, probe for what
>>                            the remote shell [default: 1]
>>                  MCA orte: parameter "orte_report_launch_progress"
>> (current value:<0>,
>>                            data source: default value)
>>                            Output a brief periodic report on launch
>> progress [default: no]
>>                  MCA orte: parameter "orte_num_boards" (current value:
>> <1>, data source:
>>                            default value)
>>                            Number of processor boards/node (1-256) [default: 
>> 1]
>>                  MCA orte: parameter "orte_num_sockets" (current value:
>> <0>, data source:
>>                            default value)
>>                            Number of sockets/board (1-256)
>>                  MCA orte: parameter "orte_num_cores" (current value:
>> <0>, data source:
>>                            default value)
>>                            Number of cores/socket (1-256)
>>                  MCA orte: parameter "orte_cpu_set" (current value:
>> <none>, data source:
>>                            default value)
>>                            Comma-separated list of ranges specifying logical 
>> cpus
>>                            allocated to this job [default: none]
>>                  MCA orte: parameter "orte_process_binding" (current
>> value:<none>, data
>>                            source: default value)
>>                            Policy for binding processes [none | core |
>> socket | board]
>>                            (supported qualifier: if-avail)
>>                  MCA orte: parameter "orte_report_bindings" (current
>> value:<0>, data
>>                            source: default value)
>>                            Report bindings
>>                  MCA orte: parameter "orte_report_events" (current
>> value:<none>, data
>>                            source: default value)
>>                            URI to which events are to be reported
>> (default: NULL)]
>>                  MCA opal: parameter "opal_net_private_ipv4" (current value:
>>
>> <10.0.0.0/8;172.16.0.0/12;192.168.0.0/16;169.254.0.0/16>, data
>>                            source: default value)
>>                            Semicolon-delimited list of CIDR notation
>> entries specifying
>>                            what networks are considered "private"
>> (default value based on
>>                            RFC1918 and RFC3330)
>>                  MCA opal: parameter "opal_signal" (current value:
>> <6,7,8,11>, data
>>                            source: default value)
>>                            Comma-delimited list of integer signal
>> numbers to Open MPI to
>>                            attempt to intercept.  Upon receipt of the
>> intercepted signal,
>>                            Open MPI will display a stack trace and
>> abort.  Open MPI will
>>                            *not* replace signals if handlers are
>> already installed by the
>>                            time MPI_INIT is invoked.  Optionally append
>> ":complain" to any
>>                            signal number in the comma-delimited list to
>> make Open MPI
>>                            complain if it detects another signal
>> handler (and therefore
>>                            does not insert its own).
>>                  MCA opal: parameter "opal_profile" (current value:
>> <0>, data source:
>>                            default value)
>>                            Set to non-zero to profile component selections
>>                  MCA opal: parameter "opal_profile_file" (current
>> value:<none>, data
>>                            source: default value)
>>                            Name of the file containing the cluster 
>> configuration
>>                            information
>>                  MCA opal: parameter "opal_paffinity_alone" (current
>> value:<0>, data
>>                            source: default value, synonyms: 
>> mpi_paffinity_alone)
>>                            If nonzero, assume that this job is the only (set 
>> of)
>>                            process(es) running on each node and bind 
>> processes to
>>                            processors, starting with processor ID 0
>>                  MCA opal: parameter "opal_set_max_sys_limits" (current
>> value:<0>, data
>>                            source: default value)
>>                            Set to non-zero to automatically set any
>> system-imposed limits
>>                            to the maximum allowed
>>                  MCA opal: parameter "opal_event_include" (current
>> value:<poll>, data
>>                            source: default value)
>>                            Comma-delimited list of libevent subsystems
>> to use (epoll,
>>                            poll, select -- available on your platform)
>>             MCA backtrace: parameter "backtrace" (current value:
>> <none>, data source:
>>                            default value)
>>                            Default selection set of components for the
>> backtrace framework
>>                            (<none>  means use all components that can be 
>> found)
>>             MCA backtrace: parameter "backtrace_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the backtrace framework
>> (0 = no verbosity)
>>             MCA backtrace: parameter "backtrace_execinfo_priority"
>> (current value:<0>,
>>                            data source: default value)
>>            MCA memchecker: parameter "memchecker" (current value:
>> <none>, data source:
>>                            default value)
>>                            Default selection set of components for the 
>> memchecker
>>                            framework (<none>  means use all components
>> that can be found)
>>                MCA memory: parameter "memory" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> memory framework
>>                            (<none>  means use all components that can be 
>> found)
>>                MCA memory: parameter "memory_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the memory framework (0
>> = no verbosity)
>>                MCA memory: information
>> "memory_linux_ptmalloc2_available" (value:<1>,
>>                            data source: default value)
>>                            Whether ptmalloc2 support is included in
>> Open MPI or not (1 =
>>                            yes, 0 = no)
>>                MCA memory: information
>> "memory_linux_ummunotify_available" (value:<0>,
>>                            data source: default value)
>>                            Whether ummunotify support is included in
>> Open MPI or not (1 =
>>                            yes, 0 = no)
>>                MCA memory: parameter "memory_linux_ptmalloc2_enable"
>> (current value:<-1>,
>>                            data source: default value)
>>                            Whether to enable ptmalloc2 support or not
>> (negative = try to
>>                            enable, but continue even if support is not
>> available, 0 = do
>>                            not enable support, positive = try to enable
>> and fail if
>>                            support is not available)
>>                MCA memory: parameter "memory_linux_ummunotify_enable"
>> (current value:
>>                            <-1>, data source: default value)
>>                            Whether to enable ummunotify support or not
>> (negative = try to
>>                            enable, but continue even if support is not
>> available, 0 = do
>>                            not enable support, positive = try to enable
>> and fail if
>>                            support is not available)
>>                MCA memory: parameter "memory_linux_disable" (current
>> value:<0>, data
>>                            source: default value)
>>                            If this MCA parameter is set to 1 **VIA
>> ENVIRONMENT VARIABLE
>>                            ONLY*** (this MCA parameter *CANNOT* be set
>> in a file or on the
>>                            mpirun command line!), this component will
>> be disabled and will
>>                            not attempt to use either ummunotify or
>> memory hook support
>>                MCA memory: parameter "memory_linux_priority" (current
>> value:<0>, data
>>                            source: default value)
>>             MCA paffinity: parameter "paffinity_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level of the paffinity framework
>>             MCA paffinity: parameter "paffinity" (current value:
>> <none>, data source:
>>                            default value)
>>                            Default selection set of components for the
>> paffinity framework
>>                            (<none>  means use all components that can be 
>> found)
>>             MCA paffinity: parameter "paffinity_hwloc_priority"
>> (current value:<40>, data
>>                            source: default value)
>>                            Priority of the hwloc paffinity component
>>                 MCA carto: parameter "carto_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level of the carto framework
>>                 MCA carto: parameter "carto" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> carto framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA carto: parameter "carto_auto_detect_priority"
>> (current value:<11>,
>>                            data source: default value)
>>                            Priority of the auto_detect carto component
>>                 MCA carto: parameter "carto_file_path" (current value:
>> <none>, data
>>                            source: default value)
>>                            The path to the cartography file
>>                 MCA carto: parameter "carto_file_priority" (current
>> value:<10>, data
>>                            source: default value)
>>                            Priority of the file carto component
>>                 MCA shmem: parameter "shmem_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level of the shmem framework
>>                 MCA shmem: parameter "shmem" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> shmem framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA shmem: parameter "shmem_mmap_enable_nfs_warning"
>> (current value:<1>,
>>                            data source: default value)
>>                            Enable the warning emitted when Open MPI
>> detects that its
>>                            shared memory backing file is located on a
>> network filesystem
>>                            (1 = enabled, 0 = disabled).
>>                 MCA shmem: parameter "shmem_mmap_priority" (current
>> value:<50>, data
>>                            source: default value)
>>                            Priority of the mmap shmem component
>>                 MCA shmem: parameter "shmem_mmap_relocate_backing_file"
>> (current value:
>>                            <0>, data source: default value)
>>                            Whether to change the default placement of
>> backing files or not
>>                            (Negative = try to relocate backing files to
>> an area rooted at
>>                            the path specified by
>>
>> shmem_mmap_opal_shmem_mmap_backing_file_base_dir, but continue
>>                            with the default path if the relocation
>> fails, 0 = do not
>>                            relocate, Positive = same as the negative
>> option, but will fail
>>                            if the relocation fails.
>>                 MCA shmem: parameter "shmem_mmap_backing_file_base_dir"
>> (current value:
>>                            </dev/shm>, data source: default value)
>>                            Specifies where backing files will be created when
>>                            shmem_mmap_relocate_backing_file is in use.
>>                 MCA shmem: parameter "shmem_posix_priority" (current
>> value:<40>, data
>>                            source: default value)
>>                            Priority of the posix shmem component
>>                 MCA shmem: parameter "shmem_sysv_priority" (current
>> value:<30>, data
>>                            source: default value)
>>                            Priority of the sysv shmem component
>>             MCA maffinity: parameter "maffinity_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level of the maffinity framework
>>             MCA maffinity: parameter "maffinity" (current value:
>> <none>, data source:
>>                            default value)
>>                            Default selection set of components for the
>> maffinity framework
>>                            (<none>  means use all components that can be 
>> found)
>>             MCA maffinity: parameter "maffinity_first_use_priority"
>> (current value:<10>,
>>                            data source: default value)
>>                            Priority of the first_use maffinity component
>>             MCA maffinity: parameter "maffinity_hwloc_priority"
>> (current value:<40>, data
>>                            source: default value)
>>                            Priority of the hwloc maffinity component
>>                 MCA timer: parameter "timer" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> timer framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA timer: parameter "timer_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the timer framework (0 =
>> no verbosity)
>>                 MCA timer: parameter "timer_linux_priority" (current
>> value:<0>, data
>>                            source: default value)
>>               MCA sysinfo: parameter "sysinfo" (current value:<none>,
>> data source:
>>                            default value)
>>                            Default selection set of components for the
>> sysinfo framework
>>                            (<none>  means use all components that can be 
>> found)
>>               MCA sysinfo: parameter "sysinfo_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the sysinfo framework (0
>> = no verbosity)
>>               MCA sysinfo: parameter "sysinfo_linux_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                 MCA hwloc: parameter "hwloc_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level of the hwloc framework
>>                 MCA hwloc: parameter "hwloc_base_mem_alloc_policy"
>> (current value:<none>,
>>                            data source: default value)
>>                            Policy that determines how general memory
>> allocations are bound
>>                            after MPI_INIT.  A value of "none" means
>> that no memory policy
>>                            is applied.  A value of "local_only" means
>> that all memory
>>                            allocations will be restricted to the local
>> NUMA node where
>>                            each process is placed.  Note that operating
>> system paging
>>                            policies are unaffected by this setting.
>> For example, if
>>                            "local_only" is used and local NUMA node
>> memory is exhausted, a
>>                            new memory allocation may cause paging.
>>                 MCA hwloc: parameter
>> "hwloc_base_mem_bind_failure_action" (current value:
>>                            <error>, data source: default value)
>>                            What Open MPI will do if it explicitly tries
>> to bind memory to
>>                            a specific NUMA location, and fails.  Note
>> that this is a
>>                            different case than the general allocation
>> policy described by
>>                            hwloc_base_alloc_policy.  A value of "warn"
>> means that Open MPI
>>                            will warn the first time this happens, but
>> allow the job to
>>                            continue (possibly with degraded
>> performance).  A value of
>>                            "error" means that Open MPI will abort the
>> job if this happens.
>>                 MCA hwloc: parameter "hwloc" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> hwloc framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA hwloc: parameter "hwloc_hwloc132_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA dpm: parameter "dpm" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> dpm framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA dpm: parameter "dpm_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the dpm framework (0 =
>> no verbosity)
>>                   MCA dpm: parameter "dpm_orte_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                MCA pubsub: parameter "pubsub" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> pubsub framework
>>                            (<none>  means use all components that can be 
>> found)
>>                MCA pubsub: parameter "pubsub_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the pubsub framework (0
>> = no verbosity)
>>                MCA pubsub: parameter "pubsub_orte_priority" (current
>> value:<50>, data
>>                            source: default value)
>>                            Priority of the pubsub pmi component
>>             MCA allocator: parameter "allocator" (current value:
>> <none>, data source:
>>                            default value)
>>                            Default selection set of components for the
>> allocator framework
>>                            (<none>  means use all components that can be 
>> found)
>>             MCA allocator: parameter "allocator_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the allocator framework
>> (0 = no verbosity)
>>             MCA allocator: parameter "allocator_basic_priority"
>> (current value:<0>, data
>>                            source: default value)
>>             MCA allocator: parameter "allocator_bucket_num_buckets"
>> (current value:<30>,
>>                            data source: default value)
>>             MCA allocator: parameter "allocator_bucket_priority"
>> (current value:<0>, data
>>                            source: default value)
>>                  MCA coll: parameter "coll" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> coll framework
>>                            (<none>  means use all components that can be 
>> found)
>>                  MCA coll: parameter "coll_base_verbose" (current
>> value:<0>, data source:
>>                            default value)
>>                            Verbosity level for the coll framework (0 =
>> no verbosity)
>>                  MCA coll: parameter "coll_basic_priority" (current
>> value:<10>, data
>>                            source: default value)
>>                            Priority of the basic coll component
>>                  MCA coll: parameter "coll_basic_crossover" (current
>> value:<4>, data
>>                            source: default value)
>>                            Minimum number of processes in a
>> communicator before using the
>>                            logarithmic algorithms
>>                  MCA coll: parameter "coll_hierarch_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                            Priority of the hierarchical coll component
>>                  MCA coll: parameter "coll_hierarch_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Turn verbose message of the hierarchical
>> coll component on/off
>>                  MCA coll: parameter "coll_hierarch_use_rdma" (current
>> value:<0>, data
>>                            source: default value)
>>                            Switch from the send btl list used to detect
>> hierarchies to the
>>                            rdma btl list
>>                  MCA coll: parameter "coll_hierarch_ignore_sm" (current
>> value:<0>, data
>>                            source: default value)
>>                            Ignore sm protocol when detecting
>> hierarchies. Required to
>>                            enable the usage of protocol specific
>> collective operations
>>                  MCA coll: parameter "coll_hierarch_detection_alg"
>> (current value:<2>,
>>                            data source: default value)
>>                            Used to specify the algorithm for detecting
>> Hierarchy.Choose
>>                            between all or two levels of hierarchy
>>                  MCA coll: parameter "coll_hierarch_bcast_alg" (current
>> value:<4>, data
>>                            source: default value)
>>                            Used to specify the algorithm used for bcast
>> operations.
>>                  MCA coll: parameter "coll_hierarch_segment_size"
>> (current value:<32768>,
>>                            data source: default value)
>>                            Used to specify the segment size for
>> segmented algorithms.
>>                  MCA coll: parameter "coll_inter_priority" (current
>> value:<40>, data
>>                            source: default value)
>>                            Priority of the inter coll component
>>                  MCA coll: parameter "coll_inter_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Turn verbose message of the inter coll
>> component on/off
>>                  MCA coll: parameter "coll_self_priority" (current
>> value:<75>, data
>>                            source: default value)
>>                  MCA coll: parameter "coll_sm_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                            Priority of the sm coll component
>>                  MCA coll: parameter "coll_sm_control_size" (current
>> value:<4096>, data
>>                            source: default value)
>>                            Length of the control data -- should usually
>> be either the
>>                            length of a cache line on most SMPs, or the
>> size of a page on
>>                            machines that support direct memory affinity
>> page placement (in
>>                            bytes)
>>                  MCA coll: parameter "coll_sm_fragment_size" (current
>> value:<8192>, data
>>                            source: default value)
>>                            Fragment size (in bytes) used for passing
>> data through shared
>>                            memory (will be rounded up to the nearest
>> control_size size)
>>                  MCA coll: parameter "coll_sm_comm_in_use_flags"
>> (current value:<2>, data
>>                            source: default value)
>>                            Number of "in use" flags, used to mark a
>> message passing area
>>                            segment as currently being used or not (must
>> be>= 2 and<=
>>                            comm_num_segments)
>>                  MCA coll: parameter "coll_sm_comm_num_segments"
>> (current value:<8>, data
>>                            source: default value)
>>                            Number of segments in each communicator's
>> shared memory message
>>                            passing area (must be>= 2, and must be a multiple 
>> of
>>                            comm_in_use_flags)
>>                  MCA coll: parameter "coll_sm_tree_degree" (current
>> value:<4>, data
>>                            source: default value)
>>                            Degree of the tree for tree-based operations
>> (must be =>  1 and
>>                            <= min(control_size, 255))
>>                  MCA coll: parameter "coll_sm_info_num_procs" (current
>> value:<4>, data
>>                            source: default value)
>>                            Number of processes to use for the calculation of 
>> the
>>                            shared_mem_size MCA information parameter
>> (must be =>  2)
>>                  MCA coll: information "coll_sm_shared_mem_used_data"
>> (value:<548864>,
>>                            data source: default value)
>>                            Amount of shared memory used, per
>> communicator, in the shared
>>                            memory data area for info_num_procs
>> processes (in bytes)
>>                  MCA coll: parameter "coll_sync_priority" (current
>> value:<50>, data
>>                            source: default value)
>>                            Priority of the sync coll component; only 
>> relevant if
>>                            barrier_before or barrier_after is>  0
>>                  MCA coll: parameter "coll_sync_barrier_before"
>> (current value:<1000>,
>>                            data source: default value)
>>                            Do a synchronization before each Nth collective
>>                  MCA coll: parameter "coll_sync_barrier_after" (current
>> value:<0>, data
>>                            source: default value)
>>                            Do a synchronization after each Nth collective
>>                  MCA coll: parameter "coll_tuned_priority" (current
>> value:<30>, data
>>                            source: default value)
>>                            Priority of the tuned coll component
>>                  MCA coll: parameter
>> "coll_tuned_pre_allocate_memory_comm_size_limit"
>>                            (current value:<32768>, data source: default 
>> value)
>>                            Size of communicator were we stop
>> pre-allocating memory for the
>>                            fixed internal buffer used for message
>> requests etc that is
>>                            hung off the communicator data segment. I.e.
>> if you have a
>>                            100'000 nodes you might not want to
>> pre-allocate 200'000
>>                            request handle slots per communicator instance!
>>                  MCA coll: parameter "coll_tuned_init_tree_fanout"
>> (current value:<4>,
>>                            data source: default value)
>>                            Inital fanout used in the tree topologies for each
>>                            communicator. This is only an initial guess,
>> if a tuned
>>                            collective needs a different fanout for an
>> operation, it build
>>                            it dynamically. This parameter is only for
>> the first guess and
>>                            might save a little time
>>                  MCA coll: parameter "coll_tuned_init_chain_fanout"
>> (current value:<4>,
>>                            data source: default value)
>>                            Inital fanout used in the chain (fanout
>> followed by pipeline)
>>                            topologies for each communicator. This is
>> only an initial
>>                            guess, if a tuned collective needs a
>> different fanout for an
>>                            operation, it build it dynamically. This
>> parameter is only for
>>                            the first guess and might save a little time
>>                  MCA coll: parameter "coll_tuned_use_dynamic_rules"
>> (current value:<0>,
>>                            data source: default value)
>>                            Switch used to decide if we use static
>> (compiled/if statements)
>>                            or dynamic (built at runtime) decision function 
>> rules
>>                    MCA io: parameter "io_base_freelist_initial_size"
>> (current value:<16>,
>>                            data source: default value)
>>                            Initial MPI-2 IO request freelist size
>>                    MCA io: parameter "io_base_freelist_max_size"
>> (current value:<64>,
>>                            data source: default value)
>>                            Max size of the MPI-2 IO request freelist
>>                    MCA io: parameter "io_base_freelist_increment"
>> (current value:<16>,
>>                            data source: default value)
>>                            Increment size of the MPI-2 IO request freelist
>>                    MCA io: parameter "io" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> io framework
>>                            (<none>  means use all components that can be 
>> found)
>>                    MCA io: parameter "io_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the io framework (0 = no
>> verbosity)
>>                    MCA io: parameter "io_romio_priority" (current
>> value:<10>, data
>>                            source: default value)
>>                            Priority of the io romio component
>>                    MCA io: parameter "io_romio_delete_priority"
>> (current value:<10>, data
>>                            source: default value)
>>                            Delete priority of the io romio component
>>                    MCA io: information "io_romio_version" (value:<from
>> MPICH2 v1.3.1 with
>>                            an additional patch from
>> romio-ma...@mcs.anl.gov about an
>>                            attribute issue>, data source: default value)
>>                            Version of ROMIO
>>                    MCA io: information "io_romio_user_configure_params"
>> (value:<none>,
>>                            data source: default value)
>>                            User-specified command line parameters
>> passed to ROMIO's
>>                            configure script
>>                    MCA io: information
>> "io_romio_complete_configure_params" (value:<
>>                            CFLAGS='-DNDEBUG -g -O2 -finline-functions
>> -fno-strict-aliasing
>>                            -pthread' CPPFLAGS='
>>
>> -I/home/andrea/library/openmpi/openmpi-1.6/opal/mca/hwloc/hwloc132/hwloc/include
>>                              -I/usr/include/infiniband 
>> -I/usr/include/infiniband'
>>                            FFLAGS='' LDFLAGS='-Wl,--rpath
>>                            -Wl,/home/andrea/library/gcc/gcc-objects/lib64 '
>>                            --enable-shared --enable-static
>>
>> --prefix=/home/andrea/library/openmpi/openmpi-1.6-gnu-4.7
>>                            --with-mpi=open_mpi --disable-aio>, data
>> source: default value)
>>                            Complete set of command line parameters
>> passed to ROMIO's
>>                            configure script
>>                 MCA mpool: parameter "mpool" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> mpool framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA mpool: parameter "mpool_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the mpool framework (0 =
>> no verbosity)
>>                 MCA mpool: parameter "mpool_fake_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                 MCA mpool: parameter "mpool_rdma_rcache_name" (current
>> value:<vma>, data
>>                            source: default value)
>>                            The name of the registration cache the mpool
>> should use
>>                 MCA mpool: parameter "mpool_rdma_rcache_size_limit"
>> (current value:<0>,
>>                            data source: default value)
>>                            the maximum size of registration cache in
>> bytes. 0 is unlimited
>>                            (default 0)
>>                 MCA mpool: parameter "mpool_rdma_print_stats" (current
>> value:<0>, data
>>                            source: default value)
>>                            print pool usage statistics at the end of the run
>>                 MCA mpool: parameter "mpool_rdma_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                 MCA mpool: parameter "mpool_sm_allocator" (current
>> value:<bucket>, data
>>                            source: default value)
>>                            Name of allocator component to use with sm mpool
>>                 MCA mpool: parameter "mpool_sm_min_size" (current
>> value:<67108864>, data
>>                            source: default value)
>>                            Minimum size of the sm mpool shared memory file
>>                 MCA mpool: parameter "mpool_sm_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Enable verbose output for mpool sm component
>>                 MCA mpool: parameter "mpool_sm_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level of the PML framework
>>                   MCA pml: parameter "pml" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> pml framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA pml: parameter "pml_bfo_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_bfo_free_list_num" (current
>> value:<4>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_bfo_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_bfo_free_list_inc" (current
>> value:<64>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_bfo_priority" (current value:
>> <5>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_bfo_send_pipeline_depth"
>> (current value:<3>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_recv_pipeline_depth"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_rdma_put_retries_limit"
>> (current value:<5>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_max_rdma_per_request"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_max_send_per_range"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_unexpected_limit"
>> (current value:<128>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_bfo_allocator" (current
>> value:<bucket>, data
>>                            source: default value)
>>                            Name of allocator component for unexpected 
>> messages
>>                   MCA pml: parameter "pml_cm_free_list_num" (current
>> value:<4>, data
>>                            source: default value)
>>                            Initial size of request free lists
>>                   MCA pml: parameter "pml_cm_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Maximum size of request free lists
>>                   MCA pml: parameter "pml_cm_free_list_inc" (current
>> value:<64>, data
>>                            source: default value)
>>                            Number of elements to add when growing
>> request free lists
>>                   MCA pml: parameter "pml_cm_priority" (current value:
>> <10>, data source:
>>                            default value)
>>                            CM PML selection priority
>>                   MCA pml: parameter "pml_csum_free_list_num" (current
>> value:<4>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_csum_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_csum_free_list_inc" (current
>> value:<64>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_csum_send_pipeline_depth"
>> (current value:<3>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_csum_recv_pipeline_depth"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_csum_rdma_put_retries_limit"
>> (current value:
>>                            <5>, data source: default value)
>>                   MCA pml: parameter "pml_csum_max_rdma_per_request"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_csum_max_send_per_range"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_csum_unexpected_limit"
>> (current value:<128>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_csum_allocator" (current
>> value:<bucket>, data
>>                            source: default value)
>>                            Name of allocator component for unexpected 
>> messages
>>                   MCA pml: parameter "pml_csum_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_ob1_free_list_num" (current
>> value:<4>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_ob1_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_ob1_free_list_inc" (current
>> value:<64>, data
>>                            source: default value)
>>                   MCA pml: parameter "pml_ob1_priority" (current value:
>> <20>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_ob1_send_pipeline_depth"
>> (current value:<3>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_recv_pipeline_depth"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_rdma_put_retries_limit"
>> (current value:<5>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_max_rdma_per_request"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_max_send_per_range"
>> (current value:<4>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_unexpected_limit"
>> (current value:<128>,
>>                            data source: default value)
>>                   MCA pml: parameter "pml_ob1_allocator" (current
>> value:<bucket>, data
>>                            source: default value)
>>                            Name of allocator component for unexpected 
>> messages
>>                   MCA pml: parameter "pml_v_priority" (current value:
>> <-1>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_v_output" (current value:
>> <stderr>, data source:
>>                            default value)
>>                   MCA pml: parameter "pml_v_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA bml: parameter "bml" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> bml framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA bml: parameter "bml_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the bml framework (0 =
>> no verbosity)
>>                   MCA bml: parameter "bml_r2_show_unreach_errors"
>> (current value:<1>,
>>                            data source: default value)
>>                            Show error message when procs are unreachable
>>                   MCA bml: parameter "bml_r2_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                MCA rcache: parameter "rcache" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> rcache framework
>>                            (<none>  means use all components that can be 
>> found)
>>                MCA rcache: parameter "rcache_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the rcache framework (0
>> = no verbosity)
>>                MCA rcache: parameter "rcache_vma_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level of the BTL framework
>>                   MCA btl: parameter "btl" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> btl framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA btl: parameter "btl_self_free_list_num" (current
>> value:<0>, data
>>                            source: default value)
>>                            Number of fragments by default
>>                   MCA btl: parameter "btl_self_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Maximum number of fragments
>>                   MCA btl: parameter "btl_self_free_list_inc" (current
>> value:<32>, data
>>                            source: default value)
>>                            Increment by this number of fragments
>>                   MCA btl: parameter "btl_self_exclusivity" (current
>> value:<65536>, data
>>                            source: default value)
>>                            BTL exclusivity (must be>= 0)
>>                   MCA btl: parameter "btl_self_flags" (current value:
>> <10>, data source:
>>                            default value)
>>                            BTL bit flags (general flags: SEND=1, PUT=2, 
>> GET=4,
>>                            SEND_INPLACE=8, RDMA_MATCHED=64,
>> HETEROGENEOUS_RDMA=256; flags
>>                            only used by the "dr" PML (ignored by others): 
>> ACK=16,
>>                            CHECKSUM=32, RDMA_COMPLETION=128; flags only
>> used by the "bfo"
>>                            PML (ignored by others): FAILOVER_SUPPORT=512)
>>                   MCA btl: parameter "btl_self_rndv_eager_limit"
>> (current value:<131072>,
>>                            data source: default value)
>>                            Size (in bytes) of "phase 1" fragment sent
>> for all large
>>                            messages (must be>= 0 and<= eager_limit)
>>                   MCA btl: parameter "btl_self_eager_limit" (current
>> value:<131072>, data
>>                            source: default value)
>>                            Maximum size (in bytes) of "short" messages
>> (must be>= 1).
>>                   MCA btl: parameter "btl_self_max_send_size" (current
>> value:<262144>,
>>                            data source: default value)
>>                            Maximum size (in bytes) of a single "phase
>> 2" fragment of a
>>                            long message when using the pipeline
>> protocol (must be>= 1)
>>                   MCA btl: parameter
>> "btl_self_rdma_pipeline_send_length" (current value:
>>                            <2147483647>, data source: default value)
>>                            Length of the "phase 2" portion of a large
>> message (in bytes)
>>                            when using the pipeline protocol.  This part
>> of the message
>>                            will be split into fragments of size
>> max_send_size and sent
>>                            using send/receive semantics (must be>= 0;
>> only relevant when
>>                            the PUT flag is set)
>>                   MCA btl: parameter "btl_self_rdma_pipeline_frag_size"
>> (current value:
>>                            <2147483647>, data source: default value)
>>                            Maximum size (in bytes) of a single "phase
>> 3" fragment from a
>>                            long message when using the pipeline
>> protocol.  These fragments
>>                            will be sent using RDMA semantics (must be
>>> = 1; only relevant
>>                            when the PUT flag is set)
>>                   MCA btl: parameter "btl_self_min_rdma_pipeline_size"
>> (current value:
>>                            <0>, data source: default value)
>>                            Messages smaller than this size (in bytes)
>> will not use the
>>                            RDMA pipeline protocol.  Instead, they will
>> be split into
>>                            fragments of max_send_size and sent using 
>> send/receive
>>                            semantics (must be>=0, and is automatically
>> adjusted up to at
>>                            least
>> (eager_limit+btl_rdma_pipeline_send_length); only
>>                            relevant when the PUT flag is set)
>>                   MCA btl: parameter "btl_self_bandwidth" (current
>> value:<100>, data
>>                            source: default value)
>>                            Approximate maximum bandwidth of
>> interconnect(0 = auto-detect
>>                            value at run-time [not supported in all BTL
>> modules],>= 1 =
>>                            bandwidth in Mbps)
>>                   MCA btl: parameter "btl_self_latency" (current value:
>> <0>, data source:
>>                            default value)
>>                            Approximate latency of interconnect (must be>= 0)
>>                   MCA btl: parameter "btl_self_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                   MCA btl: information "btl_sm_have_knem_support"
>> (value:<0>, data
>>                            source: default value)
>>                            Whether this component supports the knem
>> Linux kernel module or
>>                            not
>>                   MCA btl: parameter "btl_sm_use_knem" (current value:
>> <0>, data source:
>>                            default value)
>>                            Whether knem support is desired or not
>> (negative = try to
>>                            enable knem support, but continue even if it
>> is not available,
>>                            0 = do not enable knem support, positive =
>> try to enable knem
>>                            support and fail if it is not available)
>>                   MCA btl: parameter "btl_sm_knem_dma_min" (current
>> value:<0>, data
>>                            source: default value)
>>                            Minimum message size (in bytes) to use the
>> knem DMA mode;
>>                            ignored if knem does not support DMA mode (0
>> = do not use the
>>                            knem DMA mode)
>>                   MCA btl: parameter "btl_sm_knem_max_simultaneous"
>> (current value:<0>,
>>                            data source: default value)
>>                            Max number of simultaneous ongoing knem
>> operations to support
>>                            (0 = do everything synchronously, which
>> probably gives the best
>>                            large message latency;>0 means to do all 
>> operations
>>                            asynchronously, which supports better
>> overlap for simultaneous
>>                            large message sends)
>>                   MCA btl: parameter "btl_sm_free_list_num" (current
>> value:<8>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_free_list_inc" (current
>> value:<64>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_max_procs" (current value:
>> <-1>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_sm_mpool" (current value:
>> <sm>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_sm_fifo_size" (current value:
>> <4096>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_num_fifos" (current value:
>> <1>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_sm_fifo_lazy_free" (current
>> value:<120>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_sm_extra_procs" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_sm_exclusivity" (current
>> value:<65535>, data
>>                            source: default value)
>>                            BTL exclusivity (must be>= 0)
>>                   MCA btl: parameter "btl_sm_flags" (current value:
>> <1>, data source:
>>                            default value)
>>                            BTL bit flags (general flags: SEND=1, PUT=2, 
>> GET=4,
>>                            SEND_INPLACE=8, RDMA_MATCHED=64,
>> HETEROGENEOUS_RDMA=256; flags
>>                            only used by the "dr" PML (ignored by others): 
>> ACK=16,
>>                            CHECKSUM=32, RDMA_COMPLETION=128; flags only
>> used by the "bfo"
>>                            PML (ignored by others): FAILOVER_SUPPORT=512)
>>                   MCA btl: parameter "btl_sm_rndv_eager_limit" (current
>> value:<4096>,
>>                            data source: default value)
>>                            Size (in bytes) of "phase 1" fragment sent
>> for all large
>>                            messages (must be>= 0 and<= eager_limit)
>>                   MCA btl: parameter "btl_sm_eager_limit" (current
>> value:<4096>, data
>>                            source: default value)
>>                            Maximum size (in bytes) of "short" messages
>> (must be>= 1).
>>                   MCA btl: parameter "btl_sm_max_send_size" (current
>> value:<32768>, data
>>                            source: default value)
>>                            Maximum size (in bytes) of a single "phase
>> 2" fragment of a
>>                            long message when using the pipeline
>> protocol (must be>= 1)
>>                   MCA btl: parameter "btl_sm_bandwidth" (current value:
>> <9000>, data
>>                            source: default value)
>>                            Approximate maximum bandwidth of
>> interconnect(0 = auto-detect
>>                            value at run-time [not supported in all BTL
>> modules],>= 1 =
>>                            bandwidth in Mbps)
>>                   MCA btl: parameter "btl_sm_latency" (current value:
>> <1>, data source:
>>                            default value)
>>                            Approximate latency of interconnect (must be>= 0)
>>                   MCA btl: parameter "btl_sm_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_tcp_links" (current value:
>> <1>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_tcp_if_include" (current
>> value:<none>, data
>>                            source: default value)
>>                            Comma-delimited list of devices or CIDR
>> notation of networks to
>>                            use for MPI communication (e.g., "eth0,eth1" or
>>                            "192.168.0.0/16,10.1.4.0/24").  Mutually
>> exclusive with
>>                            btl_tcp_if_exclude.
>>                   MCA btl: parameter "btl_tcp_if_exclude" (current
>> value:<lo,sppp>, data
>>                            source: default value)
>>                            Comma-delimited list of devices or CIDR
>> notation of networks to
>>                            NOT use for MPI communication -- all devices
>> not matching these
>>                            specifications will be used (e.g., "eth0,eth1" or
>>                            "192.168.0.0/16,10.1.4.0/24").  Mutually
>> exclusive with
>>                            btl_tcp_if_include.
>>                   MCA btl: parameter "btl_tcp_free_list_num" (current
>> value:<8>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_free_list_max" (current
>> value:<-1>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_free_list_inc" (current
>> value:<32>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_sndbuf" (current value:
>> <131072>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_rcvbuf" (current value:
>> <131072>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_endpoint_cache" (current
>> value:<30720>,
>>                            data source: default value)
>>                            The size of the internal cache for each TCP
>> connection. This
>>                            cache is used to reduce the number of
>> syscalls, by replacing
>>                            them with memcpy. Every read will read the
>> expected data plus
>>                            the amount of the endpoint_cache
>>                   MCA btl: parameter "btl_tcp_use_nagle" (current
>> value:<0>, data source:
>>                            default value)
>>                            Whether to use Nagle's algorithm or not (using 
>> Nagle's
>>                            algorithm may increase short message latency)
>>                   MCA btl: parameter "btl_tcp_port_min_v4" (current
>> value:<1024>, data
>>                            source: default value)
>>                            The minimum port where the TCP BTL will try
>> to bind (default
>>                            1024)
>>                   MCA btl: parameter "btl_tcp_port_range_v4" (current
>> value:<64511>, data
>>                            source: default value)
>>                            The number of ports where the TCP BTL will
>> try to bind (default
>>                            64511). This parameter together with the
>> port min, define a
>>                            range of ports where Open MPI will open sockets.
>>                   MCA btl: parameter "btl_tcp_exclusivity" (current
>> value:<100>, data
>>                            source: default value)
>>                            BTL exclusivity (must be>= 0)
>>                   MCA btl: parameter "btl_tcp_flags" (current value:
>> <314>, data source:
>>                            default value)
>>                            BTL bit flags (general flags: SEND=1, PUT=2, 
>> GET=4,
>>                            SEND_INPLACE=8, RDMA_MATCHED=64,
>> HETEROGENEOUS_RDMA=256; flags
>>                            only used by the "dr" PML (ignored by others): 
>> ACK=16,
>>                            CHECKSUM=32, RDMA_COMPLETION=128; flags only
>> used by the "bfo"
>>                            PML (ignored by others): FAILOVER_SUPPORT=512)
>>                   MCA btl: parameter "btl_tcp_rndv_eager_limit"
>> (current value:<65536>,
>>                            data source: default value)
>>                            Size (in bytes) of "phase 1" fragment sent
>> for all large
>>                            messages (must be>= 0 and<= eager_limit)
>>                   MCA btl: parameter "btl_tcp_eager_limit" (current
>> value:<65536>, data
>>                            source: default value)
>>                            Maximum size (in bytes) of "short" messages
>> (must be>= 1).
>>                   MCA btl: parameter "btl_tcp_max_send_size" (current
>> value:<131072>,
>>                            data source: default value)
>>                            Maximum size (in bytes) of a single "phase
>> 2" fragment of a
>>                            long message when using the pipeline
>> protocol (must be>= 1)
>>                   MCA btl: parameter
>> "btl_tcp_rdma_pipeline_send_length" (current value:
>>                            <131072>, data source: default value)
>>                            Length of the "phase 2" portion of a large
>> message (in bytes)
>>                            when using the pipeline protocol.  This part
>> of the message
>>                            will be split into fragments of size
>> max_send_size and sent
>>                            using send/receive semantics (must be>= 0;
>> only relevant when
>>                            the PUT flag is set)
>>                   MCA btl: parameter "btl_tcp_rdma_pipeline_frag_size"
>> (current value:
>>                            <2147483647>, data source: default value)
>>                            Maximum size (in bytes) of a single "phase
>> 3" fragment from a
>>                            long message when using the pipeline
>> protocol.  These fragments
>>                            will be sent using RDMA semantics (must be
>>> = 1; only relevant
>>                            when the PUT flag is set)
>>                   MCA btl: parameter "btl_tcp_min_rdma_pipeline_size"
>> (current value:<0>,
>>                            data source: default value)
>>                            Messages smaller than this size (in bytes)
>> will not use the
>>                            RDMA pipeline protocol.  Instead, they will
>> be split into
>>                            fragments of max_send_size and sent using 
>> send/receive
>>                            semantics (must be>=0, and is automatically
>> adjusted up to at
>>                            least
>> (eager_limit+btl_rdma_pipeline_send_length); only
>>                            relevant when the PUT flag is set)
>>                   MCA btl: parameter "btl_tcp_bandwidth" (current
>> value:<100>, data
>>                            source: default value)
>>                            Approximate maximum bandwidth of
>> interconnect(0 = auto-detect
>>                            value at run-time [not supported in all BTL
>> modules],>= 1 =
>>                            bandwidth in Mbps)
>>                   MCA btl: parameter "btl_tcp_latency" (current value:
>> <100>, data source:
>>                            default value)
>>                            Approximate latency of interconnect (must be>= 0)
>>                   MCA btl: parameter "btl_tcp_disable_family" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_tcp_if_seq" (current value:
>> <none>, data source:
>>                            default value)
>>                            If specified, a comma-delimited list of TCP
>> interfaces.
>>                            Interfaces will be assigned, one to each MPI
>> process, in a
>>                            round-robin fashion on each server.  For
>> example, if the list
>>                            is "eth0,eth1" and four MPI processes are
>> run on a single
>>                            server, then local ranks 0 and 2 will use
>> eth0 and local ranks
>>                            1 and 3 will use eth1.
>>                   MCA btl: parameter "btl_tcp_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA btl: parameter "btl_base_include" (current value:
>> <none>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_base_exclude" (current value:
>> <none>, data
>>                            source: default value)
>>                   MCA btl: parameter "btl_base_warn_component_unused"
>> (current value:<1>,
>>                            data source: default value)
>>                            This parameter is used to turn on warning
>> messages when certain
>>                            NICs are not used
>>                   MCA mtl: parameter "mtl" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> mtl framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA mtl: parameter "mtl_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the mtl framework (0 =
>> no verbosity)
>>                  MCA topo: parameter "topo" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> topo framework
>>                            (<none>  means use all components that can be 
>> found)
>>                  MCA topo: parameter "topo_base_verbose" (current
>> value:<0>, data source:
>>                            default value)
>>                            Verbosity level for the topo framework (0 =
>> no verbosity)
>>                  MCA topo: parameter "topo_unity_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA osc: parameter "osc" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> osc framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA osc: parameter "osc_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the osc framework (0 =
>> no verbosity)
>>                   MCA osc: parameter "osc_pt2pt_no_locks" (current
>> value:<0>, data
>>                            source: default value)
>>                            Enable optimizations available only if
>> MPI_LOCK is not used.
>>                   MCA osc: parameter "osc_pt2pt_eager_limit" (current
>> value:<16384>, data
>>                            source: default value)
>>                            Max size of eagerly sent data
>>                   MCA osc: parameter "osc_pt2pt_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA osc: parameter "osc_rdma_eager_send" (current
>> value:<1>, data
>>                            source: default value)
>>                            Attempt to start data movement during
>> communication call,
>>                            instead of at synchrnoization time.  Info
>> key of same name
>>                            overrides this value.
>>                   MCA osc: parameter "osc_rdma_use_buffers" (current
>> value:<1>, data
>>                            source: default value)
>>                            Coalesce messages during an epoch to reduce 
>> network
>>                            utilization.  Info key of same name
>> overrides this value.
>>                   MCA osc: parameter "osc_rdma_use_rdma" (current
>> value:<0>, data source:
>>                            default value)
>>                            Use real RDMA operations to transfer data.
>> Info key of same
>>                            name overrides this value.
>>                   MCA osc: parameter "osc_rdma_rdma_completion_wait"
>> (current value:<1>,
>>                            data source: default value)
>>                            Wait for all completion of rdma events before 
>> sending
>>                            acknowledgment.  Info key of same name
>> overrides this value.
>>                   MCA osc: parameter "osc_rdma_no_locks" (current
>> value:<0>, data source:
>>                            default value)
>>                            Enable optimizations available only if
>> MPI_LOCK is not used.
>>                            Info key of same name overrides this value.
>>                   MCA osc: parameter "osc_rdma_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                    MCA op: parameter "op_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level of the op framework
>>                   MCA iof: parameter "iof" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> iof framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA iof: parameter "iof_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the iof framework (0 =
>> no verbosity)
>>                   MCA iof: parameter "iof_hnp_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA iof: parameter "iof_orted_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA iof: parameter "iof_tool_priority" (current
>> value:<0>, data source:
>>                            default value)
>>                   MCA oob: parameter "oob" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> oob framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA oob: parameter "oob_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the oob framework (0 =
>> no verbosity)
>>                   MCA oob: parameter "oob_tcp_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbose level for the OOB tcp component
>>                   MCA oob: parameter "oob_tcp_peer_limit" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Maximum number of peer connections to
>> simultaneously maintain
>>                            (-1 = infinite)
>>                   MCA oob: parameter "oob_tcp_peer_retries" (current
>> value:<60>, data
>>                            source: default value)
>>                            Number of times to try shutting down a
>> connection before giving
>>                            up
>>                   MCA oob: parameter "oob_tcp_debug" (current value:
>> <0>, data source:
>>                            default value)
>>                            Enable (1) / disable (0) debugging output
>> for this component
>>                   MCA oob: parameter "oob_tcp_sndbuf" (current value:
>> <131072>, data
>>                            source: default value)
>>                            TCP socket send buffering size (in bytes)
>>                   MCA oob: parameter "oob_tcp_rcvbuf" (current value:
>> <131072>, data
>>                            source: default value)
>>                            TCP socket receive buffering size (in bytes)
>>                   MCA oob: parameter "oob_tcp_if_include" (current
>> value:<none>, data
>>                            source: default value)
>>                            Comma-delimited list of TCP interfaces to use
>>                   MCA oob: parameter "oob_tcp_if_exclude" (current
>> value:<none>, data
>>                            source: default value)
>>                            Comma-delimited list of TCP interfaces to exclude
>>                   MCA oob: parameter "oob_tcp_connect_sleep" (current
>> value:<1>, data
>>                            source: default value)
>>                            Enable (1) / disable (0) random sleep for
>> connection wireup.
>>                   MCA oob: parameter "oob_tcp_listen_mode" (current
>> value:<event>, data
>>                            source: default value)
>>                            Mode for HNP to accept incoming connections: 
>> event,
>>                            listen_thread.
>>                   MCA oob: parameter "oob_tcp_listen_thread_max_queue"
>> (current value:
>>                            <10>, data source: default value)
>>                            High water mark for queued accepted socket
>> list size.  Used
>>                            only when listen_mode is listen_thread.
>>                   MCA oob: parameter "oob_tcp_listen_thread_wait_time"
>> (current value:
>>                            <10>, data source: default value)
>>                            Time in milliseconds to wait before actively
>> checking for new
>>                            connections when listen_mode is listen_thread.
>>                   MCA oob: parameter "oob_tcp_static_ports" (current
>> value:<none>, data
>>                            source: default value)
>>                            Static ports for daemons and procs (IPv4)
>>                   MCA oob: parameter "oob_tcp_dynamic_ports" (current
>> value:<none>, data
>>                            source: default value)
>>                            Range of ports to be dynamically used by
>> daemons and procs
>>                            (IPv4)
>>                   MCA oob: parameter "oob_tcp_disable_family" (current
>> value:<0>, data
>>                            source: default value)
>>                            Disable IPv4 (4) or IPv6 (6)
>>                   MCA oob: parameter "oob_tcp_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                  MCA odls: parameter "odls_base_sigkill_timeout"
>> (current value:<1>, data
>>                            source: default value)
>>                            Time to wait for a process to die after
>> issuing a kill signal
>>                            to it
>>                  MCA odls: parameter "odls" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> odls framework
>>                            (<none>  means use all components that can be 
>> found)
>>                  MCA odls: parameter "odls_base_verbose" (current
>> value:<0>, data source:
>>                            default value)
>>                            Verbosity level for the odls framework (0 =
>> no verbosity)
>>                  MCA odls: parameter "odls_default_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ras: parameter "ras_base_display_alloc" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether to display the allocation after it
>> is determined
>>                   MCA ras: parameter "ras_base_display_devel_alloc"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether to display a developer-detail
>> allocation after it is
>>                            determined
>>                   MCA ras: parameter "ras" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> ras framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA ras: parameter "ras_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the ras framework (0 =
>> no verbosity)
>>                   MCA ras: parameter "ras_cm_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA ras: parameter "ras_loadleveler_priority"
>> (current value:<90>, data
>>                            source: default value)
>>                            Priority of the loadleveler ras component
>>                   MCA ras: parameter "ras_slurm_priority" (current
>> value:<75>, data
>>                            source: default value)
>>                            Priority of the slurm ras component
>>                 MCA rmaps: parameter "rmaps_rank_file_path" (current
>> value:<none>, data
>>                            source: default value, synonym of: orte_rankfile)
>>                            Name of the rankfile to be used for mapping
>> processes (relative
>>                            or absolute path)
>>                 MCA rmaps: parameter "rmaps_base_schedule_policy"
>> (current value:<slot>,
>>                            data source: default value)
>>                            Scheduling Policy for RMAPS. [slot
>> (alias:core) | socket |
>>                            board | node]
>>                 MCA rmaps: parameter "rmaps_base_pernode" (current
>> value:<0>, data
>>                            source: default value)
>>                            Launch one ppn as directed
>>                 MCA rmaps: parameter "rmaps_base_n_pernode" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Launch n procs/node
>>                 MCA rmaps: parameter "rmaps_base_n_perboard" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Launch n procs/board
>>                 MCA rmaps: parameter "rmaps_base_n_persocket" (current
>> value:<-1>, data
>>                            source: default value)
>>                            Launch n procs/socket
>>                 MCA rmaps: parameter "rmaps_base_loadbalance" (current
>> value:<0>, data
>>                            source: default value)
>>                            Balance total number of procs across all
>> allocated nodes
>>                 MCA rmaps: parameter "rmaps_base_cpus_per_proc"
>> (current value:<1>, data
>>                            source: default value, synonyms:
>> rmaps_base_cpus_per_rank)
>>                            Number of cpus to use for each rank [1-2**15
>> (default=1)]
>>                 MCA rmaps: parameter "rmaps_base_cpus_per_rank"
>> (current value:<1>, data
>>                            source: default value, synonym of:
>> rmaps_base_cpus_per_proc)
>>                            Number of cpus to use for each rank [1-2**15
>> (default=1)]
>>                 MCA rmaps: parameter "rmaps_base_stride" (current
>> value:<1>, data source:
>>                            default value)
>>                            When binding multiple cores to a rank, the
>> step size to use
>>                            between cores [1-2**15 (default: 1)]
>>                 MCA rmaps: parameter "rmaps_base_slot_list" (current
>> value:<none>, data
>>                            source: default value)
>>                            List of processor IDs to bind MPI processes
>> to (e.g., used in
>>                            conjunction with rank files) [default=NULL]
>>                 MCA rmaps: parameter "rmaps_base_no_schedule_local"
>> (current value:<0>,
>>                            data source: default value)
>>                            If false, allow scheduling MPI applications
>> on the same node as
>>                            mpirun (default).  If true, do not schedule any 
>> MPI
>>                            applications on the same node as mpirun
>>                 MCA rmaps: parameter "rmaps_base_no_oversubscribe"
>> (current value:<0>,
>>                            data source: default value)
>>                            If true, then do not allow oversubscription
>> of nodes - mpirun
>>                            will return an error if there aren't enough
>> nodes to launch all
>>                            processes without oversubscribing
>>                 MCA rmaps: parameter "rmaps_base_display_map" (current
>> value:<0>, data
>>                            source: default value)
>>                            Whether to display the process map after it
>> is computed
>>                 MCA rmaps: parameter "rmaps_base_display_devel_map"
>> (current value:<0>,
>>                            data source: default value)
>>                            Whether to display a developer-detail
>> process map after it is
>>                            computed
>>                 MCA rmaps: parameter "rmaps" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> rmaps framework
>>                            (<none>  means use all components that can be 
>> found)
>>                 MCA rmaps: parameter "rmaps_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the rmaps framework (0 =
>> no verbosity)
>>                 MCA rmaps: parameter "rmaps_load_balance_priority"
>> (current value:<0>,
>>                            data source: default value)
>>                 MCA rmaps: parameter "rmaps_rank_file_priority"
>> (current value:<0>, data
>>                            source: default value)
>>                 MCA rmaps: parameter "rmaps_resilient_fault_grp_file"
>> (current value:
>>                            <none>, data source: default value)
>>                            Filename that contains a description of
>> fault groups for this
>>                            system
>>                 MCA rmaps: parameter "rmaps_resilient_priority"
>> (current value:<0>, data
>>                            source: default value)
>>                 MCA rmaps: parameter "rmaps_round_robin_priority"
>> (current value:<0>,
>>                            data source: default value)
>>                 MCA rmaps: parameter "rmaps_seq_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                 MCA rmaps: parameter "rmaps_topo_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA rml: parameter "rml_wrapper" (current value:
>> <none>, data source:
>>                            default value)
>>                            Use a Wrapper component around the selected
>> RML component
>>                   MCA rml: parameter "rml" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> rml framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA rml: parameter "rml_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the rml framework (0 =
>> no verbosity)
>>                   MCA rml: parameter "rml_oob_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                MCA routed: parameter "routed" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> routed framework
>>                            (<none>  means use all components that can be 
>> found)
>>                MCA routed: parameter "routed_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the routed framework (0
>> = no verbosity)
>>                MCA routed: parameter "routed_binomial_priority"
>> (current value:<0>, data
>>                            source: default value)
>>                MCA routed: parameter "routed_cm_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                MCA routed: parameter "routed_direct_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                MCA routed: parameter "routed_linear_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                MCA routed: parameter "routed_radix_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                MCA routed: parameter "routed_slave_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA plm: parameter "plm_rsh_agent" (current value:
>> <ssh : rsh>, data
>>                            source: default value, deprecated, synonym
>> of: orte_rsh_agent)
>>                            The command used to launch executables on remote 
>> nodes
>>                            (typically either "ssh" or "rsh")
>>                   MCA plm: parameter "plm_rsh_assume_same_shell"
>> (current value:<1>, data
>>                            source: default value, deprecated, synonym of:
>>                            orte_assume_same_shell)
>>                            If set to 1, assume that the shell on the
>> remote node is the
>>                            same as the shell on the local node.
>> Otherwise, probe for what
>>                            the remote shell [default: 1]
>>                   MCA plm: parameter "plm" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> plm framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA plm: parameter "plm_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the plm framework (0 =
>> no verbosity)
>>                   MCA plm: parameter "plm_rsh_num_concurrent" (current
>> value:<128>, data
>>                            source: default value)
>>                            How many plm_rsh_agent instances to invoke
>> concurrently (must
>>                            be>  0)
>>                   MCA plm: parameter "plm_rsh_force_rsh" (current
>> value:<0>, data source:
>>                            default value)
>>                            Force the launcher to always use rsh
>>                   MCA plm: parameter "plm_rsh_disable_qrsh" (current
>> value:<0>, data
>>                            source: default value)
>>                            Disable the launcher to use qrsh when under
>> the SGE parallel
>>                            environment
>>                   MCA plm: parameter "plm_rsh_daemonize_qrsh" (current
>> value:<0>, data
>>                            source: default value)
>>                            Daemonize the orted under the SGE parallel 
>> environment
>>                   MCA plm: parameter "plm_rsh_disable_llspawn" (current
>> value:<0>, data
>>                            source: default value)
>>                            Disable the use of llspawn when under the 
>> LoadLeveler
>>                            environment
>>                   MCA plm: parameter "plm_rsh_daemonize_llspawn"
>> (current value:<0>, data
>>                            source: default value)
>>                            Daemonize the orted when under the
>> LoadLeveler environment
>>                   MCA plm: parameter "plm_rsh_priority" (current value:
>> <10>, data source:
>>                            default value)
>>                            Priority of the rsh plm component
>>                   MCA plm: parameter "plm_rsh_delay" (current value:
>> <1>, data source:
>>                            default value)
>>                            Delay (in seconds) between invocations of
>> the remote agent, but
>>                            only used when the "debug" MCA parameter is
>> true, or the
>>                            top-level MCA debugging is enabled
>> (otherwise this value is
>>                            ignored)
>>                   MCA plm: parameter "plm_rsh_tree_spawn" (current
>> value:<0>, data
>>                            source: default value)
>>                            If set to 1, launch via a tree-based topology
>>                   MCA plm: parameter "plm_slurm_args" (current value:
>> <none>, data source:
>>                            default value)
>>                            Custom arguments to srun
>>                   MCA plm: parameter "plm_slurm_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                 MCA filem: parameter "filem" (current value:<none>,
>> data source: default
>>                            value)
>>                            Which Filem component to use (empty = auto-select)
>>                 MCA filem: parameter "filem_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the filem framework (0 =
>> no verbosity)
>>                 MCA filem: parameter "filem_rsh_priority" (current
>> value:<20>, data
>>                            source: default value)
>>                            Priority of the FILEM rsh component
>>                 MCA filem: parameter "filem_rsh_verbose" (current
>> value:<0>, data source:
>>                            default value)
>>                            Verbose level for the FILEM rsh component
>>                 MCA filem: parameter "filem_rsh_rcp" (current value:
>> <scp>, data source:
>>                            default value)
>>                            The rsh cp command for the FILEM rsh component
>>                 MCA filem: parameter "filem_rsh_cp" (current value:
>> <cp>, data source:
>>                            default value)
>>                            The Unix cp command for the FILEM rsh component
>>                 MCA filem: parameter "filem_rsh_rsh" (current value:
>> <ssh>, data source:
>>                            default value)
>>                            The remote shell command for the FILEM rsh 
>> component
>>                 MCA filem: parameter "filem_rsh_max_incomming" (current
>> value:<10>, data
>>                            source: default value)
>>                            Maximum number of incomming connections (0 = any)
>>                 MCA filem: parameter "filem_rsh_max_outgoing" (current
>> value:<10>, data
>>                            source: default value)
>>                            Maximum number of out going connections (0 = any)
>>                MCA errmgr: parameter "errmgr" (current value:<none>,
>> data source: default
>>                            value)
>>                            Default selection set of components for the
>> errmgr framework
>>                            (<none>  means use all components that can be 
>> found)
>>                MCA errmgr: parameter "errmgr_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the errmgr framework (0
>> = no verbosity)
>>                MCA errmgr: parameter "errmgr_default_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ess: parameter "ess" (current value:<none>, data
>> source: default
>>                            value)
>>                            Default selection set of components for the
>> ess framework
>>                            (<none>  means use all components that can be 
>> found)
>>                   MCA ess: parameter "ess_base_verbose" (current value:
>> <0>, data source:
>>                            default value)
>>                            Verbosity level for the ess framework (0 =
>> no verbosity)
>>                   MCA ess: parameter "ess_env_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA ess: parameter "ess_hnp_priority" (current value:
>> <0>, data source:
>>                            default value)
>>                   MCA ess: parameter "ess_singleton_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ess: parameter "ess_slave_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ess: parameter "ess_slurm_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ess: parameter "ess_slurmd_priority" (current
>> value:<0>, data
>>                            source: default value)
>>                   MCA ess: parameter "ess_tool_priority" (current
>> value:<0>, data source:
>>                            default value)
>>               MCA grpcomm: parameter "grpcomm" (current value:<none>,
>> data source:
>>                            default value)
>>                            Default selection set of components for the
>> grpcomm framework
>>                            (<none>  means use all components that can be 
>> found)
>>               MCA grpcomm: parameter "grpcomm_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the grpcomm framework (0
>> = no verbosity)
>>               MCA grpcomm: parameter "grpcomm_bad_priority" (current
>> value:<0>, data
>>                            source: default value)
>>               MCA grpcomm: parameter "grpcomm_basic_priority" (current
>> value:<0>, data
>>                            source: default value)
>>               MCA grpcomm: parameter "grpcomm_hier_priority" (current
>> value:<0>, data
>>                            source: default value)
>>              MCA notifier: parameter "notifier_threshold_severity"
>> (current value:
>>                            <critical>, data source: default value)
>>                            Report all events at or above this severity
>> [default: critical]
>>              MCA notifier: parameter "notifier" (current value:<none>,
>> data source:
>>                            default value)
>>                            Default selection set of components for the
>> notifier framework
>>                            (<none>  means use all components that can be 
>> found)
>>              MCA notifier: parameter "notifier_base_verbose" (current
>> value:<0>, data
>>                            source: default value)
>>                            Verbosity level for the notifier framework
>> (0 = no verbosity)
>>              MCA notifier: parameter "notifier_command_cmd" (current
>> value:</sbin/initlog
>>                            -f $s -n "Open MPI" -s "$S: $m (errorcode:
>> $e)">, data source:
>>                            default value)
>>                            Command to execute, with substitution.  $s =
>> integer severity;
>>                            $S = string severity; $e = integer error
>> code; $m = string
>>                            message
>>              MCA notifier: parameter "notifier_command_timeout"
>> (current value:<30>, data
>>                            source: default value)
>>                            Timeout (in seconds) of the command
>>              MCA notifier: parameter "notifier_command_priority"
>> (current value:<10>,
>>                            data source: default value)
>>                            Priority of this component
>>              MCA notifier: parameter "notifier_syslog_priority"
>> (current value:<0>, data
>>                            source: default value)
>>
>> ====================================================================================================
>>
>> output of cat /proc/cpuinfo
>> processor     : 0
>> vendor_id     : AuthenticAMD
>> cpu family    : 15
>> model         : 75
>> model name    : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
>> stepping      : 2
>> cpu MHz               : 1002.094
>> cache size    : 512 KB
>> physical id   : 0
>> siblings      : 2
>> core id               : 0
>> cpu cores     : 2
>> fpu           : yes
>> fpu_exception : yes
>> cpuid level   : 1
>> wp            : yes
>> flags         : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
>> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext
>> 3dnow pni cx16
>> bogomips      : 2003.90
>> TLB size      : 1088 4K pages
>> clflush size  : 64
>> cache_alignment       : 64
>> address sizes : 40 bits physical, 48 bits virtual
>> power management: ts fid vid ttp [4] [5]
>>
>> processor     : 1
>> vendor_id     : AuthenticAMD
>> cpu family    : 15
>> model         : 75
>> model name    : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
>> stepping      : 2
>> cpu MHz               : 1002.094
>> cache size    : 512 KB
>> physical id   : 0
>> siblings      : 2
>> core id               : 1
>> cpu cores     : 2
>> fpu           : yes
>> fpu_exception : yes
>> cpuid level   : 1
>> wp            : yes
>> flags         : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
>> pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext
>> 3dnow pni cx16
>> bogomips      : 2003.90
>> TLB size      : 1088 4K pages
>> clflush size  : 64
>> cache_alignment       : 64
>> address sizes : 40 bits physical, 48 bits virtual
>> power management: ts fid vid ttp [4] [5]
>>
>>
>> ====================================================================================================
>> output of ifconfig -a from a compute node
>> eth0      Link encap:Ethernet  HWaddr 00:18:F3:3F:84:A1
>>            inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
>>            inet6 addr: fe80::218:f3ff:fe3f:84a1/64 Scope:Link
>>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>            RX packets:2006 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:2064 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:1000
>>            RX bytes:242685 (236.9 KiB)  TX bytes:0 (0.0 b)
>>            Interrupt:11 Base address:0x8000
>>
>> lo        Link encap:Local Loopback
>>            inet addr:127.0.0.1  Mask:255.0.0.0
>>            inet6 addr: ::1/128 Scope:Host
>>            UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>            RX packets:60 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:0
>>            RX bytes:4440 (4.3 KiB)  TX bytes:4440 (4.3 KiB)
>>
>> sit0      Link encap:IPv6-in-IPv4
>>            NOARP  MTU:1480  Metric:1
>>            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:0
>>            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>
>>
>> ====================================================================================================
>> output of ifconfig -a from the login node, where I run mpirun
>>
>> eth0      Link encap:Ethernet  HWaddr 00:18:F3:51:B3:6E
>>            inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
>>            inet6 addr: fe80::218:f3ff:fe51:b36e/64 Scope:Link
>>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>            RX packets:7180758 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:4989496 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:1000
>>            RX bytes:6045614452 (5.6 GiB)  TX bytes:0 (0.0 b)
>>            Interrupt:201 Base address:0xe000
>>
>> eth1      Link encap:Ethernet  HWaddr 00:01:02:13:AA:3C
>>            inet addr:137.204.66.188  Bcast:137.204.66.255  Mask:255.255.255.0
>>            inet6 addr: fe80::201:2ff:fe13:aa3c/64 Scope:Link
>>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>            RX packets:4750212 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:405027 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:1000
>>            RX bytes:629146679 (600.0 MiB)  TX bytes:332118265 (316.7 MiB)
>>            Interrupt:177 Base address:0x9c00
>>
>> lo        Link encap:Local Loopback
>>            inet addr:127.0.0.1  Mask:255.0.0.0
>>            inet6 addr: ::1/128 Scope:Host
>>            UP LOOPBACK RUNNING  MTU:16436  Metric:1
>>            RX packets:288455 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:288455 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:0
>>            RX bytes:35908038 (34.2 MiB)  TX bytes:35908038 (34.2 MiB)
>>
>> sit0      Link encap:IPv6-in-IPv4
>>            NOARP  MTU:1480  Metric:1
>>            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:0
>>            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>
>> ====================================================================================================
>> output of mpirun --bynode --hostfile my_hostfile.txt --tag-output
>> ompi_info -v ompi full --parsable
>>
>> [1,0]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,0]<stdout>:ompi:version:full:1.6
>> [1,0]<stdout>:ompi:version:svn:r26429
>> [1,0]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,0]<stdout>:orte:version:full:1.6
>> [1,0]<stdout>:orte:version:svn:r26429
>> [1,0]<stdout>:orte:version:release_date:May 10, 2012
>> [1,0]<stdout>:opal:version:full:1.6
>> [1,0]<stdout>:opal:version:svn:r26429
>> [1,0]<stdout>:opal:version:release_date:May 10, 2012
>> [1,0]<stdout>:mpi-api:version:full:2.1
>> [1,0]<stdout>:ident:1.6
>> [1,6]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,6]<stdout>:ompi:version:full:1.6
>> [1,6]<stdout>:ompi:version:svn:r26429
>> [1,6]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,6]<stdout>:orte:version:full:1.6
>> [1,6]<stdout>:orte:version:svn:r26429
>> [1,6]<stdout>:orte:version:release_date:May 10, 2012
>> [1,6]<stdout>:opal:version:full:1.6
>> [1,6]<stdout>:opal:version:svn:r26429
>> [1,6]<stdout>:opal:version:release_date:May 10, 2012
>> [1,6]<stdout>:mpi-api:version:full:2.1
>> [1,6]<stdout>:ident:1.6
>> [1,9]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,10]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,3]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,3]<stdout>:ompi:version:full:1.6
>> [1,3]<stdout>:ompi:version:svn:r26429
>> [1,3]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,3]<stdout>:orte:version:full:1.6
>> [1,3]<stdout>:orte:version:svn:r26429
>> [1,3]<stdout>:orte:version:release_date:May 10, 2012
>> [1,3]<stdout>:opal:version:full:1.6
>> [1,3]<stdout>:opal:version:svn:r26429
>> [1,3]<stdout>:opal:version:release_date:May 10, 2012
>> [1,3]<stdout>:mpi-api:version:full:2.1
>> [1,3]<stdout>:ident:1.6
>> [1,4]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,4]<stdout>:ompi:version:full:1.6
>> [1,4]<stdout>:ompi:version:svn:r26429
>> [1,4]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,4]<stdout>:orte:version:full:1.6
>> [1,4]<stdout>:orte:version:svn:r26429
>> [1,4]<stdout>:orte:version:release_date:May 10, 2012
>> [1,4]<stdout>:opal:version:full:1.6
>> [1,9]<stdout>:ompi:version:full:1.6
>> [1,4]<stdout>:opal:version:svn:r26429
>> [1,4]<stdout>:opal:version:release_date:May 10, 2012
>> [1,4]<stdout>:mpi-api:version:full:2.1
>> [1,4]<stdout>:ident:1.6
>> [1,9]<stdout>:ompi:version:svn:r26429
>> [1,10]<stdout>:ompi:version:full:1.6
>> [1,9]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,10]<stdout>:ompi:version:svn:r26429
>> [1,9]<stdout>:orte:version:full:1.6
>> [1,10]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,9]<stdout>:orte:version:svn:r26429
>> [1,10]<stdout>:orte:version:full:1.6
>> [1,10]<stdout>:orte:version:svn:r26429
>> [1,9]<stdout>:orte:version:release_date:May 10, 2012
>> [1,10]<stdout>:orte:version:release_date:May 10, 2012
>> [1,9]<stdout>:opal:version:full:1.6
>> [1,10]<stdout>:opal:version:full:1.6
>> [1,9]<stdout>:opal:version:svn:r26429
>> [1,10]<stdout>:opal:version:svn:r26429
>> [1,9]<stdout>:opal:version:release_date:May 10, 2012
>> [1,10]<stdout>:opal:version:release_date:May 10, 2012
>> [1,9]<stdout>:mpi-api:version:full:2.1
>> [1,9]<stdout>:ident:1.6
>> [1,10]<stdout>:mpi-api:version:full:2.1
>> [1,10]<stdout>:ident:1.6
>> [1,2]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,2]<stdout>:ompi:version:full:1.6
>> [1,2]<stdout>:ompi:version:svn:r26429
>> [1,2]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,2]<stdout>:orte:version:full:1.6
>> [1,2]<stdout>:orte:version:svn:r26429
>> [1,2]<stdout>:orte:version:release_date:May 10, 2012
>> [1,2]<stdout>:opal:version:full:1.6
>> [1,2]<stdout>:opal:version:svn:r26429
>> [1,2]<stdout>:opal:version:release_date:May 10, 2012
>> [1,2]<stdout>:mpi-api:version:full:2.1
>> [1,2]<stdout>:ident:1.6
>> [1,8]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,8]<stdout>:ompi:version:full:1.6
>> [1,8]<stdout>:ompi:version:svn:r26429
>> [1,8]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,8]<stdout>:orte:version:full:1.6
>> [1,8]<stdout>:orte:version:svn:r26429
>> [1,8]<stdout>:orte:version:release_date:May 10, 2012
>> [1,8]<stdout>:opal:version:full:1.6
>> [1,8]<stdout>:opal:version:svn:r26429
>> [1,8]<stdout>:opal:version:release_date:May 10, 2012
>> [1,8]<stdout>:mpi-api:version:full:2.1
>> [1,8]<stdout>:ident:1.6
>> [1,11]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,11]<stdout>:ompi:version:full:1.6
>> [1,11]<stdout>:ompi:version:svn:r26429
>> [1,11]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,11]<stdout>:orte:version:full:1.6
>> [1,11]<stdout>:orte:version:svn:r26429
>> [1,11]<stdout>:orte:version:release_date:May 10, 2012
>> [1,11]<stdout>:opal:version:full:1.6
>> [1,11]<stdout>:opal:version:svn:r26429
>> [1,11]<stdout>:opal:version:release_date:May 10, 2012
>> [1,11]<stdout>:mpi-api:version:full:2.1
>> [1,11]<stdout>:ident:1.6
>> [1,5]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,5]<stdout>:ompi:version:full:1.6
>> [1,5]<stdout>:ompi:version:svn:r26429
>> [1,5]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,5]<stdout>:orte:version:full:1.6
>> [1,5]<stdout>:orte:version:svn:r26429
>> [1,5]<stdout>:orte:version:release_date:May 10, 2012
>> [1,5]<stdout>:opal:version:full:1.6
>> [1,5]<stdout>:opal:version:svn:r26429
>> [1,5]<stdout>:opal:version:release_date:May 10, 2012
>> [1,5]<stdout>:mpi-api:version:full:2.1
>> [1,5]<stdout>:ident:1.6
>> [1,1]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,7]<stdout>:package:Open MPI and...@cloud.bo.astro.it Distribution
>> [1,7]<stdout>:ompi:version:full:1.6
>> [1,7]<stdout>:ompi:version:svn:r26429
>> [1,7]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,7]<stdout>:orte:version:full:1.6
>> [1,7]<stdout>:orte:version:svn:r26429
>> [1,7]<stdout>:orte:version:release_date:May 10, 2012
>> [1,7]<stdout>:opal:version:full:1.6
>> [1,7]<stdout>:opal:version:svn:r26429
>> [1,7]<stdout>:opal:version:release_date:May 10, 2012
>> [1,7]<stdout>:mpi-api:version:full:2.1
>> [1,7]<stdout>:ident:1.6
>> [1,1]<stdout>:ompi:version:full:1.6
>> [1,1]<stdout>:ompi:version:svn:r26429
>> [1,1]<stdout>:ompi:version:release_date:May 10, 2012
>> [1,1]<stdout>:orte:version:full:1.6
>> [1,1]<stdout>:orte:version:svn:r26429
>> [1,1]<stdout>:orte:version:release_date:May 10, 2012
>> [1,1]<stdout>:opal:version:full:1.6
>> [1,1]<stdout>:opal:version:svn:r26429
>> [1,1]<stdout>:opal:version:release_date:May 10, 2012
>> [1,1]<stdout>:mpi-api:version:full:2.1
>> [1,1]<stdout>:ident:1.6
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ------------------------------
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> End of users Digest, Vol 2339, Issue 5
> **************************************

Reply via email to