On Thu, Jul 12, 2012 at 10:53:52AM -0400, Jonathan Dursi wrote: > Hi: > > One of our users is reporting trouble reading large files with > MPI_File_read (or read_all). With a few different type sizes, to > keep count lower than 2^31, the problem persists. A simple C > program to test this is attached; we see it in both OpenMPI 1.4.4 > and OpenMPI 1.6, with the only difference being the error code > returned. We can read the amount of data required by looping over > MPI_File_read()s, but in more complicated scenarios this gets > awkward. I always thought that the 32-bit signed count limitation > wasn't so bad because you could create larger data types to get > around it, but this appears not to be the case here. Is this a > known problem that we should just work around?
Hi. Known problem in the ROMIO MPI-IO implementation (which OpenMPI uses). Been on my list of "things to fix" for a while. ==rob > Output from ompi_info --all for the 1.4.4 build is also attached. > > > OpenMPI 1.4.4 > > Trying 268435457 of float, 1073741828 bytes: successfully read 268435457 > Trying 536870913 of float, 2147483652 bytes: failed: err=35, MPI_ERR_IO: > input/output error > Trying 134217729 of double, 1073741832 bytes: successfully read 134217729 > Trying 268435457 of double, 2147483656 bytes: failed: err=35, MPI_ERR_IO: > input/output error > Trying 67108865 of 2xdouble, 1073741840 bytes: successfully read 67108865 > Trying 134217729 of 2xdouble, 2147483664 bytes: failed: err=35, MPI_ERR_IO: > input/output error > Trying 524289 of 256xdouble, 1073743872 bytes: successfully read 524289 > Trying 1048577 of 256xdouble, 2147485696 bytes: failed: err=35, MPI_ERR_IO: > input/output error > Chunk 1/2: Trying 524288 of 256xdouble, chunked, 1073741824 bytes: > successfully read 524288 > Chunk 2/2: Trying 524289 of 256xdouble, chunked, 1073743872 bytes: > successfully read 524289 > > > OpenMPI 1.6 > Trying 268435457 of float, 1073741828 bytes: successfully read 268435457 > Trying 536870913 of float, 2147483652 bytes: failed: err=13, MPI_ERR_ARG: > invalid argument of some other kind > Trying 134217729 of double, 1073741832 bytes: successfully read 134217729 > Trying 268435457 of double, 2147483656 bytes: failed: err=13, MPI_ERR_ARG: > invalid argument of some other kind > Trying 67108865 of 2xdouble, 1073741840 bytes: successfully read 67108865 > Trying 134217729 of 2xdouble, 2147483664 bytes: failed: err=13, MPI_ERR_ARG: > invalid argument of some other kind > Trying 524289 of 256xdouble, 1073743872 bytes: successfully read 524289 > Trying 1048577 of 256xdouble, 2147485696 bytes: failed: err=13, MPI_ERR_ARG: > invalid argument of some other kind > Chunk 1/2: Trying 524288 of 256xdouble, chunked, 1073741824 bytes: > successfully read 524288 > Chunk 2/2: Trying 524289 of 256xdouble, chunked, 1073743872 bytes: > successfully read 524289 > > - Jonathan > > > #include <stdio.h> > #include <stdlib.h> > #include <mpi.h> > > int tryToRead(const MPI_File file, const MPI_Datatype type, const int count, > const size_t size, const char *typename, void *buf) { > > int ierr; > MPI_Status status; > > size_t bufsize = (size_t)count * size; > > printf("Trying %d of %s, %lu bytes: ", count, typename, bufsize ); > > ierr = MPI_File_read(file, buf, count, type, &status); > > if ( !ierr ) { > > int gotcount; > MPI_Get_count( &status, type, &gotcount ); > printf("successfully read %d\n", gotcount); > > } else { > > char err[MPI_MAX_ERROR_STRING]; > int len; > MPI_Error_string(ierr, err, &len); > printf("failed: err=%d, %s\n", ierr, err); > > } > > return ierr; > } > > > int tryToReadInChunks(const MPI_File file, const MPI_Datatype type, const int > count, > const size_t size, const char *typename, void *buf, int > nchunks) { > > int ierr; > int nsofar = 0; > int chunksize = count / nchunks; > char *cbuf = buf; > > for (int chunk = 0; chunk < nchunks; chunk++ ) { > int thischunk = chunksize; > if (chunk == nchunks-1) thischunk = count - nsofar; > > printf("Chunk %d/%d: ", chunk+1, nchunks); > ierr = tryToRead(file, type, thischunk, size, typename, > &(cbuf[nsofar*size])); > > if (ierr) break; > nsofar += thischunk; > } > > return ierr; > } > > > int main(int argc, char *argv[]) > { > int count; > MPI_File fh; > > MPI_Init(&argc, &argv); > > MPI_File_open(MPI_COMM_WORLD, "/dev/zero", MPI_MODE_RDONLY, > MPI_INFO_NULL, &fh); > > char *buf = malloc( ((size_t)1 << 31)+1024 ); > if (buf == NULL) { > printf("Malloc failed.\n"); > exit(-1); > } > > /* integers */ > > count = (1 << 28) + 1; > tryToRead(fh, MPI_FLOAT, count, sizeof(float), "float", buf); > > count = (1 << 29) + 1; > tryToRead(fh, MPI_FLOAT, count, sizeof(float), "float", buf); > > /* doubles */ > > count = (1 << 27) + 1; > tryToRead(fh, MPI_DOUBLE, count, sizeof(double), "double", buf); > > count = (1 << 28) + 1; > tryToRead(fh, MPI_DOUBLE, count, sizeof(double), "double", buf); > > /* 2 x doubles */ > > MPI_Datatype TwoDoubles; > MPI_Type_contiguous(2, MPI_DOUBLE, &TwoDoubles); > MPI_Type_commit(&TwoDoubles); > > count = (1 << 26) + 1; > tryToRead(fh, TwoDoubles, count, 2*sizeof(double), "2xdouble", buf); > > count = (1 << 27) + 1; > tryToRead(fh, TwoDoubles, count, 2*sizeof(double), "2xdouble", buf); > > MPI_Type_free(&TwoDoubles); > > > /* 256 x doubles */ > > MPI_Datatype TwoFiftySixDoubles; > MPI_Type_contiguous(256, MPI_DOUBLE, &TwoFiftySixDoubles); > MPI_Type_commit(&TwoFiftySixDoubles); > > count = (1 << 19) + 1; > tryToRead(fh, TwoFiftySixDoubles, count, 256*sizeof(double), > "256xdouble", buf); > > count = (1 << 20) + 1; > tryToRead(fh, TwoFiftySixDoubles, count, 256*sizeof(double), > "256xdouble", buf); > > > /* 256 x doubles, in 2 chunks */ > > count = (1 << 20) + 1; > tryToReadInChunks(fh, TwoFiftySixDoubles, count, 256*sizeof(double), > "256xdouble, chunked", buf, 2); > > MPI_Type_free(&TwoFiftySixDoubles); > > MPI_File_close(&fh); > MPI_Finalize(); > > free(buf); > return 0; > } > > > Package: Open MPI root@gpc-f109n010 Distribution > Open MPI: 1.4.4 > Open MPI SVN revision: r25188 > Open MPI release date: Sep 27, 2011 > Open RTE: 1.4.4 > Open RTE SVN revision: r25188 > Open RTE release date: Sep 27, 2011 > OPAL: 1.4.4 > OPAL SVN revision: r25188 > OPAL release date: Sep 27, 2011 > Ident string: 1.4.4 > MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.4.4) > MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.4) > MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.4) > MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.4.4) > MCA carto: file (MCA v2.0, API v2.0, Component v1.4.4) > MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.4) > MCA timer: linux (MCA v2.0, API v2.0, Component v1.4.4) > MCA installdirs: env (MCA v2.0, API v2.0, Component v1.4.4) > MCA installdirs: config (MCA v2.0, API v2.0, Component v1.4.4) > MCA dpm: orte (MCA v2.0, API v2.0, Component v1.4.4) > MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.4.4) > MCA allocator: basic (MCA v2.0, API v2.0, Component v1.4.4) > MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: basic (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: inter (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: self (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: sm (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: sync (MCA v2.0, API v2.0, Component v1.4.4) > MCA coll: tuned (MCA v2.0, API v2.0, Component v1.4.4) > MCA io: romio (MCA v2.0, API v2.0, Component v1.4.4) > MCA mpool: fake (MCA v2.0, API v2.0, Component v1.4.4) > MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.4.4) > MCA mpool: sm (MCA v2.0, API v2.0, Component v1.4.4) > MCA pml: cm (MCA v2.0, API v2.0, Component v1.4.4) > MCA pml: csum (MCA v2.0, API v2.0, Component v1.4.4) > MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.4.4) > MCA pml: v (MCA v2.0, API v2.0, Component v1.4.4) > MCA bml: r2 (MCA v2.0, API v2.0, Component v1.4.4) > MCA rcache: vma (MCA v2.0, API v2.0, Component v1.4.4) > MCA btl: self (MCA v2.0, API v2.0, Component v1.4.4) > MCA btl: sm (MCA v2.0, API v2.0, Component v1.4.4) > MCA btl: ofud (MCA v2.0, API v2.0, Component v1.4.4) > MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.4) > MCA btl: tcp (MCA v2.0, API v2.0, Component v1.4.4) > MCA topo: unity (MCA v2.0, API v2.0, Component v1.4.4) > MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.4.4) > MCA osc: rdma (MCA v2.0, API v2.0, Component v1.4.4) > MCA iof: hnp (MCA v2.0, API v2.0, Component v1.4.4) > MCA iof: orted (MCA v2.0, API v2.0, Component v1.4.4) > MCA iof: tool (MCA v2.0, API v2.0, Component v1.4.4) > MCA oob: tcp (MCA v2.0, API v2.0, Component v1.4.4) > MCA odls: default (MCA v2.0, API v2.0, Component v1.4.4) > MCA ras: slurm (MCA v2.0, API v2.0, Component v1.4.4) > MCA ras: tm (MCA v2.0, API v2.0, Component v1.4.4) > MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.4.4) > MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.4.4) > MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.4.4) > MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.4.4) > MCA rml: oob (MCA v2.0, API v2.0, Component v1.4.4) > MCA routed: binomial (MCA v2.0, API v2.0, Component v1.4.4) > MCA routed: direct (MCA v2.0, API v2.0, Component v1.4.4) > MCA routed: linear (MCA v2.0, API v2.0, Component v1.4.4) > MCA plm: rsh (MCA v2.0, API v2.0, Component v1.4.4) > MCA plm: slurm (MCA v2.0, API v2.0, Component v1.4.4) > MCA plm: tm (MCA v2.0, API v2.0, Component v1.4.4) > MCA filem: rsh (MCA v2.0, API v2.0, Component v1.4.4) > MCA errmgr: default (MCA v2.0, API v2.0, Component v1.4.4) > MCA ess: env (MCA v2.0, API v2.0, Component v1.4.4) > MCA ess: hnp (MCA v2.0, API v2.0, Component v1.4.4) > MCA ess: singleton (MCA v2.0, API v2.0, Component v1.4.4) > MCA ess: slurm (MCA v2.0, API v2.0, Component v1.4.4) > MCA ess: tool (MCA v2.0, API v2.0, Component v1.4.4) > MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.4.4) > MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.4.4) > Prefix: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1 > Exec_prefix: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1 > Bindir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/bin > Sbindir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/sbin > Libdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/lib > Incdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/include > Mandir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/man > Pkglibdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/lib/openmpi > Libexecdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/libexec > Datarootdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share > Datadir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share > Sysconfdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/etc > Sharedstatedir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/com > Localstatedir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/var > Infodir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/info > Pkgdatadir: > /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/openmpi > Pkglibdir: /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/lib/openmpi > Pkgincludedir: > /scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/include/openmpi > Configured architecture: x86_64-unknown-linux-gnu > Configure host: gpc-f109n010 > Configured by: root > Configured on: Fri Nov 25 14:12:53 EST 2011 > Configure host: gpc-f109n010 > Built by: root > Built on: Fri Nov 25 14:23:02 EST 2011 > Built host: gpc-f109n010 > C bindings: yes > C++ bindings: yes > Fortran77 bindings: yes (all) > Fortran90 bindings: yes > Fortran90 bindings size: small > C compiler: gcc > C compiler absolute: /scinet/gpc/compilers/gcc-4.6.1/bin/gcc > C char size: 1 > C bool size: 1 > C short size: 2 > C int size: 4 > C long size: 8 > C float size: 4 > C double size: 8 > C pointer size: 8 > C char align: 1 > C bool align: 1 > C int align: 4 > C float align: 4 > C double align: 8 > C++ compiler: g++ > C++ compiler absolute: /scinet/gpc/compilers/gcc-4.6.1/bin/g++ > Fortran77 compiler: gfortran > Fortran77 compiler abs: /scinet/gpc/compilers/gcc-4.6.1/bin/gfortran > Fortran90 compiler: gfortran > Fortran90 compiler abs: /scinet/gpc/compilers/gcc-4.6.1/bin/gfortran > Fort integer size: 4 > Fort logical size: 4 > Fort logical value true: 1 > Fort have integer1: yes > Fort have integer2: yes > Fort have integer4: yes > Fort have integer8: yes > Fort have integer16: no > Fort have real4: yes > Fort have real8: yes > Fort have real16: no > Fort have complex8: yes > Fort have complex16: yes > Fort have complex32: no > Fort integer1 size: 1 > Fort integer2 size: 2 > Fort integer4 size: 4 > Fort integer8 size: 8 > Fort integer16 size: -1 > Fort real size: 4 > Fort real4 size: 4 > Fort real8 size: 8 > Fort real16 size: 16 > Fort dbl prec size: 4 > Fort cplx size: 4 > Fort dbl cplx size: 4 > Fort cplx8 size: 8 > Fort cplx16 size: 16 > Fort cplx32 size: 32 > Fort integer align: 4 > Fort integer1 align: 1 > Fort integer2 align: 2 > Fort integer4 align: 4 > Fort integer8 align: 8 > Fort integer16 align: -1 > Fort real align: 4 > Fort real4 align: 4 > Fort real8 align: 8 > Fort real16 align: 16 > Fort dbl prec align: 4 > Fort cplx align: 4 > Fort dbl cplx align: 4 > Fort cplx8 align: 4 > Fort cplx16 align: 8 > Fort cplx32 align: 16 > C profiling: yes > C++ profiling: yes > Fortran77 profiling: yes > Fortran90 profiling: yes > C++ exceptions: no > Thread support: posix (mpi: yes, progress: no) > Sparse Groups: no > Build CFLAGS: -O3 -DNDEBUG -finline-functions > -fno-strict-aliasing -pthread -fvisibility=hidden > Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread > Build FFLAGS: > Build FCFLAGS: > Build LDFLAGS: -export-dynamic > Build LIBS: -lnsl -lutil -lm > Wrapper extra CFLAGS: -pthread > Wrapper extra CXXFLAGS: -pthread > Wrapper extra FFLAGS: -pthread > Wrapper extra FCFLAGS: -pthread > Wrapper extra LDFLAGS: -L/opt/torque/lib -Wl,--rpath -Wl,/opt/torque/lib > > Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -ldl > -Wl,--export-dynamic -lnsl -lutil -lm -ldl > Internal debug support: no > MPI parameter check: runtime > Memory profiling support: no > Memory debugging support: no > libltdl support: yes > Heterogeneous support: no > mpirun default --prefix: no > MPI I/O support: yes > MPI_WTIME support: gettimeofday > Symbol visibility support: yes > FT Checkpoint support: no (checkpoint thread: no) > MCA mca: parameter "mca_param_files" (current value: > "/home/s/scinet/ljdursi/.openmpi/mca-params.conf:/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/etc/openmpi-mca-params.conf", > data source: default value) > Path for MCA configuration files containing default > parameter values > MCA mca: parameter "mca_base_param_file_prefix" (current > value: <none>, data source: default value) > Aggregate MCA parameter file sets > MCA mca: parameter "mca_base_param_file_path" (current > value: > "/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/openmpi/amca-param-sets:/scratch/s/scinet/ljdursi/Testing/mpi", > data source: default value) > Aggregate MCA parameter Search path > MCA mca: parameter "mca_base_param_file_path_force" (current > value: <none>, data source: default value) > Forced Aggregate MCA parameter Search path > MCA mca: parameter "mca_component_path" (current value: > "/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/lib/openmpi:/home/s/scinet/ljdursi/.openmpi/components", > data source: default value) > Path where to look for Open MPI and ORTE components > MCA mca: parameter "mca_verbose" (current value: <none>, > data source: default value) > Top-level verbosity parameter > MCA mca: parameter "mca_component_show_load_errors" (current > value: "1", data source: default value) > Whether to show errors for components that failed > to load or not > MCA mca: parameter "mca_component_disable_dlopen" (current > value: "0", data source: default value) > Whether to attempt to disable opening dynamic > components or not > MCA mpi: parameter "mpi_paffinity_alone" (current value: > "0", data source: default value, synonym of: opal_paffinity_alone) > If nonzero, assume that this job is the only (set > of) process(es) running on each node and bind processes to processors, > starting with processor ID 0 > MCA mpi: parameter "mpi_param_check" (current value: "1", > data source: default value) > Whether you want MPI API parameters checked at > run-time or not. Possible values are 0 (no checking) and 1 (perform checking > at run-time) > MCA mpi: parameter "mpi_yield_when_idle" (current value: > "-1", data source: default value) > Yield the processor when waiting for MPI > communication (for MPI processes, will default to 1 when oversubscribing > nodes) > MCA mpi: parameter "mpi_event_tick_rate" (current value: > "-1", data source: default value) > How often to progress TCP communications (0 = > never, otherwise specified in microseconds) > MCA mpi: parameter "mpi_show_handle_leaks" (current value: > "0", data source: default value) > Whether MPI_FINALIZE shows all MPI handles that > were not freed or not > MCA mpi: parameter "mpi_no_free_handles" (current value: > "0", data source: default value) > Whether to actually free MPI objects when their > handles are freed > MCA mpi: parameter "mpi_show_mpi_alloc_mem_leaks" (current > value: "0", data source: default value) > If >0, MPI_FINALIZE will show up to this many > instances of memory allocated by MPI_ALLOC_MEM that was not freed by > MPI_FREE_MEM > MCA mpi: parameter "mpi_show_mca_params" (current value: > <none>, data source: default value) > Whether to show all MCA parameter values during > MPI_INIT or not (good for reproducability of MPI jobs for debug purposes). > Accepted values are all, default, file, api, and enviro - or a comma > delimited combination of them > MCA mpi: parameter "mpi_show_mca_params_file" (current > value: <none>, data source: default value) > If mpi_show_mca_params is true, setting this string > to a valid filename tells Open MPI to dump all the MCA parameter values into > a file suitable for reading via the mca_param_files parameter (good for > reproducability of MPI jobs) > MCA mpi: parameter "mpi_keep_peer_hostnames" (current value: > "1", data source: default value) > If nonzero, save the string hostnames of all MPI > peer processes (mostly for error / debugging output messages). This can add > quite a bit of memory usage to each MPI process. > MCA mpi: parameter "mpi_abort_delay" (current value: "0", > data source: default value) > If nonzero, print out an identifying message when > MPI_ABORT is invoked (hostname, PID of the process that called MPI_ABORT) and > delay for that many seconds before exiting (a negative delay value means to > never abort). This allows attaching of a debugger before quitting the job. > MCA mpi: parameter "mpi_abort_print_stack" (current value: > "0", data source: default value) > If nonzero, print out a stack trace when MPI_ABORT > is invoked > MCA mpi: parameter "mpi_preconnect_mpi" (current value: "0", > data source: default value, synonyms: mpi_preconnect_all) > Whether to force MPI processes to fully wire-up the > MPI connections between MPI processes during MPI_INIT (vs. making connections > lazily -- upon the first MPI traffic between each process peer pair) > MCA mpi: parameter "mpi_preconnect_all" (current value: "0", > data source: default value, deprecated, synonym of: mpi_preconnect_mpi) > Whether to force MPI processes to fully wire-up the > MPI connections between MPI processes during MPI_INIT (vs. making connections > lazily -- upon the first MPI traffic between each process peer pair) > MCA mpi: parameter "mpi_leave_pinned" (current value: "-1", > data source: default value) > Whether to use the "leave pinned" protocol or not. > Enabling this setting can help bandwidth performance when repeatedly sending > and receiving large messages with the same buffers over RDMA-based networks > (0 = do not use "leave pinned" protocol, 1 = use "leave pinned" protocol, -1 > = allow network to choose at runtime). > MCA mpi: parameter "mpi_leave_pinned_pipeline" (current > value: "0", data source: default value) > Whether to use the "leave pinned pipeline" protocol > or not. > MCA mpi: parameter "mpi_warn_on_fork" (current value: "1", > data source: default value) > If nonzero, issue a warning if program forks under > conditions that could cause system errors > MCA mpi: information "mpi_have_sparse_group_storage" (value: > "0", data source: default value) > Whether this Open MPI installation supports storing > of data in MPI groups in "sparse" formats (good for extremely large process > count MPI jobs that create many communicators/groups) > MCA mpi: parameter "mpi_use_sparse_group_storage" (current > value: "0", data source: default value) > Whether to use "sparse" storage formats for MPI > groups (only relevant if mpi_have_sparse_group_storage is 1) > MCA orte: parameter "orte_base_help_aggregate" (current > value: "1", data source: default value) > If orte_base_help_aggregate is true, duplicate help > messages will be aggregated rather than displayed individually. This can be > helpful for parallel jobs that experience multiple identical failures; rather > than print out the same help/failure message N times, display it once with a > count of how many processes sent the same message. > MCA orte: parameter "orte_tmpdir_base" (current value: > <none>, data source: default value) > Base of the session directory tree > MCA orte: parameter "orte_no_session_dirs" (current value: > <none>, data source: default value) > Prohibited locations for session directories > (multiple locations separated by ',', default=NULL) > MCA orte: parameter "orte_debug" (current value: "0", data > source: default value) > Top-level ORTE debug switch (default verbosity: 1) > MCA orte: parameter "orte_debug_verbose" (current value: > "-1", data source: default value) > Verbosity level for ORTE debug messages (default: 1) > MCA orte: parameter "orte_debug_daemons" (current value: "0", > data source: default value) > Whether to debug the ORTE daemons or not > MCA orte: parameter "orte_debug_daemons_file" (current value: > "0", data source: default value) > Whether want stdout/stderr of daemons to go to a > file or not > MCA orte: parameter "orte_leave_session_attached" (current > value: "0", data source: default value) > Whether applications and/or daemons should leave > their sessions attached so that any output can be received - this allows X > forwarding without all the attendant debugging output > MCA orte: parameter "orte_do_not_launch" (current value: "0", > data source: default value) > Perform all necessary operations to prepare to > launch the application, but do not actually launch it > MCA orte: parameter "orte_daemon_spin" (current value: "0", > data source: default value) > Have any orteds spin until we can connect a > debugger to them > MCA orte: parameter "orte_daemon_fail" (current value: "-1", > data source: default value) > Have the specified orted fail after init for > debugging purposes > MCA orte: parameter "orte_daemon_fail_delay" (current value: > "0", data source: default value) > Have the specified orted fail after specified > number of seconds (default: 0 => no delay) > MCA orte: parameter "orte_heartbeat_rate" (current value: > "0", data source: default value) > Seconds between checks for daemon state-of-health > (default: 0 => do not check) > MCA orte: parameter "orte_startup_timeout" (current value: > "0", data source: default value) > Milliseconds/daemon to wait for startup before > declaring failed_to_start (default: 0 => do not check) > MCA orte: parameter "orte_timing" (current value: "0", data > source: default value) > Request that critical timing loops be measured > MCA orte: parameter "orte_base_user_debugger" (current value: > "totalview @mpirun@ -a @mpirun_args@ : ddt -n @np@ -start @executable@ > @executable_argv@ @single_app@ : fxp @mpirun@ -a @mpirun_args@", data source: > default value) > Sequence of user-level debuggers to search for in > orterun > MCA orte: parameter "orte_abort_timeout" (current value: "1", > data source: default value) > Max time to wait [in secs] before aborting an ORTE > operation (default: 1sec) > MCA orte: parameter "orte_timeout_step" (current value: > "1000", data source: default value) > Time to wait [in usecs/proc] before aborting an > ORTE operation (default: 1000 usec/proc) > MCA orte: parameter "orte_default_hostfile" (current value: > <none>, data source: default value) > Name of the default hostfile (relative or absolute > path) > MCA orte: parameter "orte_rankfile" (current value: <none>, > data source: default value, synonyms: rmaps_rank_file_path) > Name of the rankfile to be used for mapping > processes (relative or absolute path) > MCA orte: parameter "orte_keep_fqdn_hostnames" (current > value: "0", data source: default value) > Whether or not to keep FQDN hostnames [default: no] > MCA orte: parameter "orte_contiguous_nodes" (current value: > "2147483647", data source: default value) > Number of nodes after which contiguous nodename > encoding will automatically be used [default: INT_MAX] > MCA orte: parameter "orte_tag_output" (current value: "0", > data source: default value) > Tag all output with [job,rank] (default: false) > MCA orte: parameter "orte_xml_output" (current value: "0", > data source: default value) > Display all output in XML format (default: false) > MCA orte: parameter "orte_xml_file" (current value: <none>, > data source: default value) > Provide all output in XML format to the specified > file > MCA orte: parameter "orte_timestamp_output" (current value: > "0", data source: default value) > Timestamp all application process output (default: > false) > MCA orte: parameter "orte_output_filename" (current value: > <none>, data source: default value) > Redirect output from application processes into > filename.rank [default: NULL] > MCA orte: parameter "orte_show_resolved_nodenames" (current > value: "0", data source: default value) > Display any node names that are resolved to a > different name (default: false) > MCA orte: parameter "orte_hetero_apps" (current value: "0", > data source: default value) > Indicates that multiple app_contexts are being > provided that are a mix of 32/64 bit binaries (default: false) > MCA orte: parameter "orte_launch_agent" (current value: > "orted", data source: default value) > Command used to start processes on remote nodes > (default: orted) > MCA orte: parameter "orte_allocation_required" (current > value: "0", data source: default value) > Whether or not an allocation by a resource manager > is required [default: no] > MCA orte: parameter "orte_xterm" (current value: <none>, data > source: default value) > Create a new xterm window and display output from > the specified ranks there [default: none] > MCA orte: parameter "orte_forward_job_control" (current > value: "0", data source: default value) > Forward SIGTSTP (after converting to SIGSTOP) and > SIGCONT signals to the application procs [default: no] > MCA orte: parameter "orte_report_launch_progress" (current > value: "0", data source: default value) > Output a brief periodic report on launch progress > [default: no] > MCA orte: parameter "orte_num_boards" (current value: "1", > data source: default value) > Number of processor boards/node (1-256) [default: 1] > MCA orte: parameter "orte_num_sockets" (current value: "0", > data source: default value) > Number of sockets/board (1-256) > MCA orte: parameter "orte_num_cores" (current value: "0", > data source: default value) > Number of cores/socket (1-256) > MCA orte: parameter "orte_cpu_set" (current value: <none>, > data source: default value) > Comma-separated list of ranges specifying logical > cpus allocated to this job [default: none] > MCA orte: parameter "orte_process_binding" (current value: > <none>, data source: default value) > Policy for binding processes [none | core | socket > | board] (supported qualifier: if-avail) > MCA opal: parameter "opal_net_private_ipv4" (current value: > "10.0.0.0/8;172.16.0.0/12;192.168.0.0/16;169.254.0.0/16", data source: > default value) > Semicolon-delimited list of CIDR notation entries > specifying what networks are considered "private" (default value based on > RFC1918 and RFC3330) > MCA opal: parameter "opal_signal" (current value: "6,7,8,11", > data source: default value) > Comma-delimited list of integer signal numbers to > Open MPI to attempt to intercept. Upon receipt of the intercepted signal, > Open MPI will display a stack trace and abort. Open MPI will *not* replace > signals if handlers are already installed by the time MPI_INIT is invoked. > Optionally append ":complain" to any signal number in the comma-delimited > list to make Open MPI complain if it detects another signal handler (and > therefore does not insert its own). > MCA opal: parameter "opal_paffinity_alone" (current value: > "0", data source: default value, synonyms: mpi_paffinity_alone) > If nonzero, assume that this job is the only (set > of) process(es) running on each node and bind processes to processors, > starting with processor ID 0 > MCA opal: parameter "opal_set_max_sys_limits" (current value: > "0", data source: default value) > Set to non-zero to automatically set any > system-imposed limits to the maximum allowed > MCA opal: parameter "opal_event_include" (current value: > "poll", data source: default value) > Comma-delimited list of libevent subsystems to use > (poll, select -- available on your platform) > MCA backtrace: parameter "backtrace" (current value: <none>, data > source: default value) > Default selection set of components for the > backtrace framework (<none> means use all components that can be found) > MCA backtrace: parameter "backtrace_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the backtrace framework (0 = no > verbosity) > MCA backtrace: parameter "backtrace_execinfo_priority" (current > value: "0", data source: default value) > MCA memchecker: parameter "memchecker" (current value: <none>, data > source: default value) > Default selection set of components for the > memchecker framework (<none> means use all components that can be found) > MCA memory: parameter "memory" (current value: <none>, data > source: default value) > Default selection set of components for the memory > framework (<none> means use all components that can be found) > MCA memory: parameter "memory_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the memory framework (0 = no > verbosity) > MCA memory: parameter "memory_ptmalloc2_priority" (current > value: "0", data source: default value) > MCA paffinity: parameter "paffinity_base_verbose" (current value: > "0", data source: default value) > Verbosity level of the paffinity framework > MCA paffinity: parameter "paffinity" (current value: <none>, data > source: default value) > Default selection set of components for the > paffinity framework (<none> means use all components that can be found) > MCA paffinity: parameter "paffinity_linux_priority" (current > value: "10", data source: default value) > Priority of the linux paffinity component > MCA paffinity: information "paffinity_linux_plpa_version" (value: > "1.3.2", data source: default value) > Version of PLPA that is embedded in Open MPI > MCA carto: parameter "carto_base_verbose" (current value: "0", > data source: default value) > Verbosity level of the carto framework > MCA carto: parameter "carto" (current value: <none>, data > source: default value) > Default selection set of components for the carto > framework (<none> means use all components that can be found) > MCA carto: parameter "carto_auto_detect_priority" (current > value: "11", data source: default value) > Priority of the auto_detect carto component > MCA carto: parameter "carto_file_path" (current value: <none>, > data source: default value) > The path to the cartography file > MCA carto: parameter "carto_file_priority" (current value: > "10", data source: default value) > Priority of the file carto component > MCA maffinity: parameter "maffinity_base_verbose" (current value: > "0", data source: default value) > Verbosity level of the maffinity framework > MCA maffinity: parameter "maffinity" (current value: <none>, data > source: default value) > Default selection set of components for the > maffinity framework (<none> means use all components that can be found) > MCA maffinity: parameter "maffinity_first_use_priority" (current > value: "10", data source: default value) > Priority of the first_use maffinity component > MCA timer: parameter "timer" (current value: <none>, data > source: default value) > Default selection set of components for the timer > framework (<none> means use all components that can be found) > MCA timer: parameter "timer_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the timer framework (0 = no > verbosity) > MCA timer: parameter "timer_linux_priority" (current value: > "0", data source: default value) > MCA dpm: parameter "dpm" (current value: <none>, data > source: default value) > Default selection set of components for the dpm > framework (<none> means use all components that can be found) > MCA dpm: parameter "dpm_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the dpm framework (0 = no > verbosity) > MCA pubsub: parameter "pubsub" (current value: <none>, data > source: default value) > Default selection set of components for the pubsub > framework (<none> means use all components that can be found) > MCA pubsub: parameter "pubsub_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the pubsub framework (0 = no > verbosity) > MCA pubsub: parameter "pubsub_orte_priority" (current value: > "0", data source: default value) > MCA allocator: parameter "allocator" (current value: <none>, data > source: default value) > Default selection set of components for the > allocator framework (<none> means use all components that can be found) > MCA allocator: parameter "allocator_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the allocator framework (0 = no > verbosity) > MCA allocator: parameter "allocator_basic_priority" (current > value: "0", data source: default value) > MCA allocator: parameter "allocator_bucket_num_buckets" (current > value: "30", data source: default value) > MCA allocator: parameter "allocator_bucket_priority" (current > value: "0", data source: default value) > MCA coll: parameter "coll" (current value: <none>, data > source: default value) > Default selection set of components for the coll > framework (<none> means use all components that can be found) > MCA coll: parameter "coll_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the coll framework (0 = no > verbosity) > MCA coll: parameter "coll_basic_priority" (current value: > "10", data source: default value) > Priority of the basic coll component > MCA coll: parameter "coll_basic_crossover" (current value: > "4", data source: default value) > Minimum number of processes in a communicator > before using the logarithmic algorithms > MCA coll: parameter "coll_hierarch_priority" (current value: > "0", data source: default value) > Priority of the hierarchical coll component > MCA coll: parameter "coll_hierarch_verbose" (current value: > "0", data source: default value) > Turn verbose message of the hierarchical coll > component on/off > MCA coll: parameter "coll_hierarch_use_rdma" (current value: > "0", data source: default value) > Switch from the send btl list used to detect > hierarchies to the rdma btl list > MCA coll: parameter "coll_hierarch_ignore_sm" (current value: > "0", data source: default value) > Ignore sm protocol when detecting hierarchies. > Required to enable the usage of protocol specific collective operations > MCA coll: parameter "coll_hierarch_detection_alg" (current > value: "2", data source: default value) > Used to specify the algorithm for detecting > Hierarchy.To specify all levels or two levels of hierarchy > MCA coll: parameter "coll_inter_priority" (current value: > "40", data source: default value) > Priority of the inter coll component > MCA coll: parameter "coll_inter_verbose" (current value: "0", > data source: default value) > Turn verbose message of the inter coll component > on/off > MCA coll: parameter "coll_self_priority" (current value: > "75", data source: default value) > MCA coll: parameter "coll_sm_priority" (current value: "0", > data source: default value) > Priority of the sm coll component > MCA coll: parameter "coll_sm_control_size" (current value: > "4096", data source: default value) > Length of the control data -- should usually be > either the length of a cache line on most SMPs, or the size of a page on > machines that support direct memory affinity page placement (in bytes) > MCA coll: parameter "coll_sm_fragment_size" (current value: > "8192", data source: default value) > Fragment size (in bytes) used for passing data > through shared memory (will be rounded up to the nearest control_size size) > MCA coll: parameter "coll_sm_comm_in_use_flags" (current > value: "2", data source: default value) > Number of "in use" flags, used to mark a message > passing area segment as currently being used or not (must be >= 2 and <= > comm_num_segments) > MCA coll: parameter "coll_sm_comm_num_segments" (current > value: "128", data source: default value) > Number of segments in each communicator's shared > memory message passing area (must be >= 2, and must be a multiple of > comm_in_use_flags) > MCA coll: parameter "coll_sm_tree_degree" (current value: > "4", data source: default value) > Degree of the tree for tree-based operations (must > be => 1 and <= min(control_size, 255)) > MCA coll: parameter "coll_sm_info_num_procs" (current value: > "4", data source: default value) > Number of processes to use for the calculation of > the shared_mem_size MCA information parameter (must be => 2) > MCA coll: information "coll_sm_shared_mem_used_data" (value: > "8413184", data source: default value) > Amount of shared memory used, per communicator, in > the shared memory data area for info_num_procs processes (in bytes) > MCA coll: parameter "coll_sync_priority" (current value: > "50", data source: default value) > Priority of the sync coll component; only relevant > if barrier_before or barrier_after is > 0 > MCA coll: parameter "coll_sync_barrier_before" (current > value: "1000", data source: default value) > Do a synchronization before each Nth collective > MCA coll: parameter "coll_sync_barrier_after" (current value: > "0", data source: default value) > Do a synchronization after each Nth collective > MCA coll: parameter "coll_tuned_priority" (current value: > "30", data source: default value) > Priority of the tuned coll component > MCA coll: parameter > "coll_tuned_pre_allocate_memory_comm_size_limit" (current value: "32768", > data source: default value) > Size of communicator were we stop pre-allocating > memory for the fixed internal buffer used for message requests etc that is > hung off the communicator data segment. I.e. if you have a 100'000 nodes you > might not want to pre-allocate 200'000 request handle slots per communicator > instance! > MCA coll: parameter "coll_tuned_init_tree_fanout" (current > value: "4", data source: default value) > Inital fanout used in the tree topologies for each > communicator. This is only an initial guess, if a tuned collective needs a > different fanout for an operation, it build it dynamically. This parameter is > only for the first guess and might save a little time > MCA coll: parameter "coll_tuned_init_chain_fanout" (current > value: "4", data source: default value) > Inital fanout used in the chain (fanout followed by > pipeline) topologies for each communicator. This is only an initial guess, if > a tuned collective needs a different fanout for an operation, it build it > dynamically. This parameter is only for the first guess and might save a > little time > MCA coll: parameter "coll_tuned_use_dynamic_rules" (current > value: "0", data source: default value) > Switch used to decide if we use static (compiled/if > statements) or dynamic (built at runtime) decision function rules > MCA io: parameter "io_base_freelist_initial_size" (current > value: "16", data source: default value) > Initial MPI-2 IO request freelist size > MCA io: parameter "io_base_freelist_max_size" (current > value: "64", data source: default value) > Max size of the MPI-2 IO request freelist > MCA io: parameter "io_base_freelist_increment" (current > value: "16", data source: default value) > Increment size of the MPI-2 IO request freelist > MCA io: parameter "io" (current value: <none>, data source: > default value) > Default selection set of components for the io > framework (<none> means use all components that can be found) > MCA io: parameter "io_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the io framework (0 = no > verbosity) > MCA io: parameter "io_romio_priority" (current value: "10", > data source: default value) > Priority of the io romio component > MCA io: parameter "io_romio_delete_priority" (current > value: "10", data source: default value) > Delete priority of the io romio component > MCA io: information "io_romio_version" (value: "from MPICH2 > v1.0.7 with additional compilation/bug patches from romio-ma...@mcs.anl.gov", > data source: default value) > Version of ROMIO > MCA io: information "io_romio_user_configure_params" > (value: <none>, data source: default value) > User-specified command line parameters passed to > ROMIO's configure script > MCA io: information "io_romio_complete_configure_params" > (value: " CFLAGS='-O3 -DNDEBUG -finline-functions -fno-strict-aliasing > -pthread' CPPFLAGS=' ' FFLAGS='' LDFLAGS=' ' --enable-shared --enable-static > --prefix=/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1 --with-mpi=open_mpi > --disable-aio", data source: default value) > Complete set of command line parameters passed to > ROMIO's configure script > MCA mpool: parameter "mpool" (current value: <none>, data > source: default value) > Default selection set of components for the mpool > framework (<none> means use all components that can be found) > MCA mpool: parameter "mpool_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the mpool framework (0 = no > verbosity) > MCA mpool: parameter "mpool_fake_priority" (current value: > "0", data source: default value) > MCA mpool: parameter "mpool_rdma_rcache_name" (current value: > "vma", data source: default value) > The name of the registration cache the mpool should > use > MCA mpool: parameter "mpool_rdma_rcache_size_limit" (current > value: "0", data source: default value) > the maximum size of registration cache in bytes. 0 > is unlimited (default 0) > MCA mpool: parameter "mpool_rdma_print_stats" (current value: > "0", data source: default value) > print pool usage statistics at the end of the run > MCA mpool: parameter "mpool_rdma_priority" (current value: > "0", data source: default value) > MCA mpool: parameter "mpool_sm_allocator" (current value: > "bucket", data source: default value) > Name of allocator component to use with sm mpool > MCA mpool: parameter "mpool_sm_min_size" (current value: > "67108864", data source: default value) > Minimum size of the sm mpool shared memory file > MCA mpool: parameter "mpool_sm_verbose" (current value: "0", > data source: default value) > Enable verbose output for mpool sm component > MCA mpool: parameter "mpool_sm_priority" (current value: "0", > data source: default value) > MCA pml: parameter "pml_base_verbose" (current value: "0", > data source: default value) > Verbosity level of the PML framework > MCA pml: parameter "pml" (current value: <none>, data > source: default value) > Default selection set of components for the pml > framework (<none> means use all components that can be found) > MCA pml: parameter "pml_cm_free_list_num" (current value: > "4", data source: default value) > Initial size of request free lists > MCA pml: parameter "pml_cm_free_list_max" (current value: > "-1", data source: default value) > Maximum size of request free lists > MCA pml: parameter "pml_cm_free_list_inc" (current value: > "64", data source: default value) > Number of elements to add when growing request free > lists > MCA pml: parameter "pml_cm_priority" (current value: "30", > data source: default value) > CM PML selection priority > MCA pml: parameter "pml_csum_free_list_num" (current value: > "4", data source: default value) > MCA pml: parameter "pml_csum_free_list_max" (current value: > "-1", data source: default value) > MCA pml: parameter "pml_csum_free_list_inc" (current value: > "64", data source: default value) > MCA pml: parameter "pml_csum_send_pipeline_depth" (current > value: "3", data source: default value) > MCA pml: parameter "pml_csum_recv_pipeline_depth" (current > value: "4", data source: default value) > MCA pml: parameter "pml_csum_rdma_put_retries_limit" > (current value: "5", data source: default value) > MCA pml: parameter "pml_csum_max_rdma_per_request" (current > value: "4", data source: default value) > MCA pml: parameter "pml_csum_max_send_per_range" (current > value: "4", data source: default value) > MCA pml: parameter "pml_csum_unexpected_limit" (current > value: "128", data source: default value) > MCA pml: parameter "pml_csum_allocator" (current value: > "bucket", data source: default value) > Name of allocator component for unexpected messages > MCA pml: parameter "pml_csum_priority" (current value: "0", > data source: default value) > MCA pml: parameter "pml_ob1_free_list_num" (current value: > "4", data source: default value) > MCA pml: parameter "pml_ob1_free_list_max" (current value: > "-1", data source: default value) > MCA pml: parameter "pml_ob1_free_list_inc" (current value: > "64", data source: default value) > MCA pml: parameter "pml_ob1_priority" (current value: "20", > data source: default value) > MCA pml: parameter "pml_ob1_send_pipeline_depth" (current > value: "3", data source: default value) > MCA pml: parameter "pml_ob1_recv_pipeline_depth" (current > value: "4", data source: default value) > MCA pml: parameter "pml_ob1_rdma_put_retries_limit" (current > value: "5", data source: default value) > MCA pml: parameter "pml_ob1_max_rdma_per_request" (current > value: "4", data source: default value) > MCA pml: parameter "pml_ob1_max_send_per_range" (current > value: "4", data source: default value) > MCA pml: parameter "pml_ob1_unexpected_limit" (current > value: "128", data source: default value) > MCA pml: parameter "pml_ob1_allocator" (current value: > "bucket", data source: default value) > Name of allocator component for unexpected messages > MCA pml: parameter "pml_v_priority" (current value: "-1", > data source: default value) > MCA pml: parameter "pml_v_output" (current value: "stderr", > data source: default value) > MCA pml: parameter "pml_v_verbose" (current value: "0", data > source: default value) > MCA bml: parameter "bml" (current value: <none>, data > source: default value) > Default selection set of components for the bml > framework (<none> means use all components that can be found) > MCA bml: parameter "bml_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the bml framework (0 = no > verbosity) > MCA bml: parameter "bml_r2_show_unreach_errors" (current > value: "1", data source: default value) > Show error message when procs are unreachable > MCA bml: parameter "bml_r2_priority" (current value: "0", > data source: default value) > MCA rcache: parameter "rcache" (current value: <none>, data > source: default value) > Default selection set of components for the rcache > framework (<none> means use all components that can be found) > MCA rcache: parameter "rcache_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the rcache framework (0 = no > verbosity) > MCA rcache: parameter "rcache_vma_priority" (current value: > "0", data source: default value) > MCA btl: parameter "btl_base_verbose" (current value: "0", > data source: default value) > Verbosity level of the BTL framework > MCA btl: parameter "btl" (current value: <none>, data > source: default value) > Default selection set of components for the btl > framework (<none> means use all components that can be found) > MCA btl: parameter "btl_self_free_list_num" (current value: > "0", data source: default value) > Number of fragments by default > MCA btl: parameter "btl_self_free_list_max" (current value: > "-1", data source: default value) > Maximum number of fragments > MCA btl: parameter "btl_self_free_list_inc" (current value: > "32", data source: default value) > Increment by this number of fragments > MCA btl: parameter "btl_self_exclusivity" (current value: > "65536", data source: default value) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_self_flags" (current value: "10", > data source: default value) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128) > MCA btl: parameter "btl_self_rndv_eager_limit" (current > value: "131072", data source: default value) > Size (in bytes) of "phase 1" fragment sent for all > large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_self_eager_limit" (current value: > "131072", data source: default value) > Maximum size (in bytes) of "short" messages (must > be >= 1). > MCA btl: parameter "btl_self_max_send_size" (current value: > "262144", data source: default value) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_self_rdma_pipeline_send_length" > (current value: "2147483647", data source: default value) > Length of the "phase 2" portion of a large message > (in bytes) when using the pipeline protocol. This part of the message will > be split into fragments of size max_send_size and sent using send/receive > semantics (must be >= 0; only relevant when the PUT flag is set) > MCA btl: parameter "btl_self_rdma_pipeline_frag_size" > (current value: "2147483647", data source: default value) > Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant when > the PUT flag is set) > MCA btl: parameter "btl_self_min_rdma_pipeline_size" > (current value: "0", data source: default value) > Messages smaller than this size (in bytes) will not > use the RDMA pipeline protocol. Instead, they will be split into fragments > of max_send_size and sent using send/receive semantics (must be >=0, and is > automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag > is set) > MCA btl: parameter "btl_self_bandwidth" (current value: > "100", data source: default value) > Approximate maximum bandwidth of interconnect(must > be >= 1) > MCA btl: parameter "btl_self_latency" (current value: "0", > data source: default value) > Approximate latency of interconnect (must be >= 0) > MCA btl: parameter "btl_self_priority" (current value: "0", > data source: default value) > MCA btl: parameter "btl_sm_free_list_num" (current value: > "8", data source: default value) > MCA btl: parameter "btl_sm_free_list_max" (current value: > "-1", data source: default value) > MCA btl: parameter "btl_sm_free_list_inc" (current value: > "64", data source: default value) > MCA btl: parameter "btl_sm_max_procs" (current value: "-1", > data source: default value) > MCA btl: parameter "btl_sm_mpool" (current value: "sm", data > source: default value) > MCA btl: parameter "btl_sm_fifo_size" (current value: > "4096", data source: default value) > MCA btl: parameter "btl_sm_num_fifos" (current value: "1", > data source: default value) > MCA btl: parameter "btl_sm_fifo_lazy_free" (current value: > "120", data source: default value) > MCA btl: parameter "btl_sm_sm_extra_procs" (current value: > "0", data source: default value) > MCA btl: parameter "btl_sm_exclusivity" (current value: > "65535", data source: default value) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_sm_flags" (current value: "1", data > source: default value) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128) > MCA btl: parameter "btl_sm_rndv_eager_limit" (current value: > "4096", data source: default value) > Size (in bytes) of "phase 1" fragment sent for all > large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_sm_eager_limit" (current value: > "4096", data source: default value) > Maximum size (in bytes) of "short" messages (must > be >= 1). > MCA btl: parameter "btl_sm_max_send_size" (current value: > "32768", data source: default value) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_sm_bandwidth" (current value: "900", > data source: default value) > Approximate maximum bandwidth of interconnect(must > be >= 1) > MCA btl: parameter "btl_sm_latency" (current value: "100", > data source: default value) > Approximate latency of interconnect (must be >= 0) > MCA btl: parameter "btl_sm_priority" (current value: "0", > data source: default value) > MCA btl: parameter "btl_ofud_max_btls" (current value: "4", > data source: default value) > Maximum number of HCAs/ports to use > MCA btl: parameter "btl_ofud_mpool" (current value: "rdma", > data source: default value) > Name of the memory pool to be used > MCA btl: parameter "btl_ofud_ib_pkey_index" (current value: > "0", data source: default value) > IB pkey index > MCA btl: parameter "btl_ofud_ib_qkey" (current value: > "20119859", data source: default value) > IB qkey > MCA btl: parameter "btl_ofud_ib_service_level" (current > value: "0", data source: default value) > IB service level > MCA btl: parameter "btl_ofud_ib_src_path_bits" (current > value: "0", data source: default value) > IB source path bits > MCA btl: parameter "btl_ofud_sd_num" (current value: "128", > data source: default value) > maximum send descriptors to post > MCA btl: parameter "btl_ofud_rd_num" (current value: "6000", > data source: default value) > number of receive buffers > MCA btl: parameter "btl_ofud_min_send_size" (current value: > "2048", data source: default value) > minimum send size > MCA btl: parameter "btl_ofud_max_send_size" (current value: > "2048", data source: default value) > maximum send size > MCA btl: parameter "btl_ofud_exclusivity" (current value: > "1024", data source: default value) > BTL exclusivity > MCA btl: parameter "btl_ofud_bandwidth" (current value: > "800", data source: default value) > Approximate maximum bandwidth of interconnect > MCA btl: parameter "btl_ofud_priority" (current value: "0", > data source: default value) > MCA btl: parameter "btl_openib_verbose" (current value: "0", > data source: default value) > Output some verbose OpenIB BTL information (0 = no > output, nonzero = output) > MCA btl: parameter "btl_openib_warn_no_device_params_found" > (current value: "1", data source: default value, synonyms: > btl_openib_warn_no_hca_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > MCA btl: parameter "btl_openib_warn_no_hca_params_found" > (current value: "1", data source: default value, deprecated, synonym of: > btl_openib_warn_no_device_params_found) > Warn when no device-specific parameters are found > in the INI file specified by the btl_openib_device_param_files MCA parameter > (0 = do not warn; any other value = warn) > MCA btl: parameter "btl_openib_warn_default_gid_prefix" > (current value: "1", data source: default value) > Warn when there is more than one active ports and > at least one of them connected to the network with only default GID prefix > configured (0 = do not warn; any other value = warn) > MCA btl: parameter "btl_openib_warn_nonexistent_if" (current > value: "1", data source: default value) > Warn if non-existent devices and/or ports are > specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn; > any other value = warn) > MCA btl: parameter "btl_openib_want_fork_support" (current > value: "-1", data source: default value) > Whether fork support is desired or not (negative = > try to enable fork support, but continue even if it is not available, 0 = do > not enable fork support, positive = try to enable fork support and fail if it > is not available) > MCA btl: parameter "btl_openib_device_param_files" (current > value: > "/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/openmpi/mca-btl-openib-device-params.ini", > data source: default value, synonyms: btl_openib_hca_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters > MCA btl: parameter "btl_openib_hca_param_files" (current > value: > "/scinet/gpc/mpi/openmpi/1.4.4-gcc-v4.6.1/share/openmpi/mca-btl-openib-device-params.ini", > data source: default value, deprecated, synonym of: > btl_openib_device_param_files) > Colon-delimited list of INI-style files that > contain device vendor/part-specific parameters > MCA btl: parameter "btl_openib_device_type" (current value: > "all", data source: default value) > Specify to only use IB or iWARP network adapters > (infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = > use any available adapters) > MCA btl: parameter "btl_openib_max_btls" (current value: > "-1", data source: default value) > Maximum number of device ports to use (-1 = use all > available, otherwise must be >= 1) > MCA btl: parameter "btl_openib_free_list_num" (current > value: "8", data source: default value) > Intial size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_free_list_max" (current > value: "-1", data source: default value) > Maximum size of free lists (-1 = infinite, > otherwise must be >= 0) > MCA btl: parameter "btl_openib_free_list_inc" (current > value: "32", data source: default value) > Increment size of free lists (must be >= 1) > MCA btl: parameter "btl_openib_mpool" (current value: > "rdma", data source: default value) > Name of the memory pool to be used (it is unlikely > that you will ever want to change this > MCA btl: parameter "btl_openib_reg_mru_len" (current value: > "16", data source: default value) > Length of the registration cache most recently used > list (must be >= 1) > MCA btl: parameter "btl_openib_cq_size" (current value: > "1000", data source: default value, synonyms: btl_openib_ib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_ib_cq_size" (current value: > "1000", data source: default value, deprecated, synonym of: > btl_openib_cq_size) > Minimum size of the OpenFabrics completion queue > (CQs are automatically sized based on the number of peer MPI processes; this > value determines the *minimum* size of all CQs) > MCA btl: parameter "btl_openib_max_inline_data" (current > value: "-1", data source: default value, synonyms: > btl_openib_ib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_ib_max_inline_data" (current > value: "-1", data source: default value, deprecated, synonym of: > btl_openib_max_inline_data) > Maximum size of inline data segment (-1 = run-time > probe to discover max value, otherwise must be >= 0). If not explicitly set, > use max_inline_data from the INI file containing device-specific parameters > MCA btl: parameter "btl_openib_pkey" (current value: "0", > data source: default value, synonyms: btl_openib_ib_pkey_val) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB paritition key value (0x7fff) > MCA btl: parameter "btl_openib_ib_pkey_val" (current value: > "0", data source: default value, deprecated, synonym of: btl_openib_pkey) > OpenFabrics partition key (pkey) value. Unsigned > integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be > masked against the maximum allowable IB paritition key value (0x7fff) > MCA btl: parameter "btl_openib_psn" (current value: "0", > data source: default value, synonyms: btl_openib_ib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_psn" (current value: "0", > data source: default value, deprecated, synonym of: btl_openib_psn) > OpenFabrics packet sequence starting number (must > be >= 0) > MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current > value: "4", data source: default value) > InfiniBand outstanding atomic reads (must be >= 0) > MCA btl: parameter "btl_openib_mtu" (current value: "3", > data source: default value, synonyms: btl_openib_ib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > MCA btl: parameter "btl_openib_ib_mtu" (current value: "3", > data source: default value, deprecated, synonym of: btl_openib_mtu) > OpenFabrics MTU, in bytes (if not specified in INI > files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048 > bytes, 5=4096 bytes > MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current > value: "25", data source: default value) > InfiniBand minimum "receiver not ready" timer, in > seconds (must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_timeout" (current value: > "20", data source: default value) > InfiniBand transmit timeout, plugged into formula: > 4.096 microseconds * (2^btl_openib_ib_timeout)(must be >= 0 and <= 31) > MCA btl: parameter "btl_openib_ib_retry_count" (current > value: "7", data source: default value) > InfiniBand transmit retry count (must be >= 0 and > <= 7) > MCA btl: parameter "btl_openib_ib_rnr_retry" (current value: > "7", data source: default value) > InfiniBand "receiver not ready" retry count; > applies *only* to SRQ/XRC queues. PP queues use RNR retry values of 0 > because Open MPI performs software flow control to guarantee that RNRs never > occur (must be >= 0 and <= 7; 7 = "infinite") > MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current > value: "4", data source: default value) > InfiniBand maximum pending RDMA destination > operations (must be >= 0) > MCA btl: parameter "btl_openib_ib_service_level" (current > value: "0", data source: default value) > InfiniBand service level (must be >= 0 and <= 15) > MCA btl: parameter "btl_openib_use_eager_rdma" (current > value: "-1", data source: default value) > Use RDMA for eager messages (-1 = use device > default, 0 = do not use eager RDMA, 1 = use eager RDMA) > MCA btl: parameter "btl_openib_eager_rdma_threshold" > (current value: "16", data source: default value) > Use RDMA for short messages after this number of > messages are received from a given peer (must be >= 1) > MCA btl: parameter "btl_openib_max_eager_rdma" (current > value: "16", data source: default value) > Maximum number of peers allowed to use RDMA for > short messages (RDMA is used for all long messages, except if explicitly > disabled, such as with the "dr" pml) (must be >= 0) > MCA btl: parameter "btl_openib_eager_rdma_num" (current > value: "16", data source: default value) > Number of RDMA buffers to allocate for small > messages(must be >= 1) > MCA btl: parameter "btl_openib_btls_per_lid" (current value: > "1", data source: default value) > Number of BTLs to create for each InfiniBand LID > (must be >= 1) > MCA btl: parameter "btl_openib_max_lmc" (current value: "0", > data source: default value) > Maximum number of LIDs to use for each device port > (must be >= 0, where 0 = use all available) > MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current > value: "0", data source: default value) > Maximum number of alterative paths for each device > port (must be >= -1, where 0 = disable apm, -1 = all availible alternative > paths ) > MCA btl: parameter "btl_openib_enable_apm_over_ports" > (current value: "0", data source: default value) > Enable alterative path migration (APM) over > different ports of the same device (must be >= 0, where 0 = disable APM over > ports , 1 = enable APM over ports of the same device) > MCA btl: parameter "btl_openib_use_async_event_thread" > (current value: "1", data source: default value) > If nonzero, use the thread that will handle > InfiniBand asyncihronous events > MCA btl: parameter "btl_openib_buffer_alignment" (current > value: "64", data source: default value) > Prefered communication buffer alignment, in bytes > (must be > 0 and power of two) > MCA btl: parameter "btl_openib_use_message_coalescing" > (current value: "1", data source: default value) > Use message coalescing > MCA btl: parameter "btl_openib_cq_poll_ratio" (current > value: "100", data source: default value) > how often poll high priority CQ versus low priority > CQ > MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" > (current value: "100", data source: default value) > how often poll eager RDMA channel versus CQ > MCA btl: parameter "btl_openib_hp_cq_poll_per_progress" > (current value: "10", data source: default value) > max number of completion events to process for each > call of BTL progress engine > MCA btl: information "btl_openib_have_fork_support" (value: > "1", data source: default value) > Whether the OpenFabrics stack supports applications > that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that > this value does NOT indicate whether the system being run on supports > "fork()" with OpenFabrics applications or not. > MCA btl: parameter "btl_openib_exclusivity" (current value: > "1024", data source: default value) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_openib_flags" (current value: "310", > data source: default value) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128) > MCA btl: parameter "btl_openib_rndv_eager_limit" (current > value: "12288", data source: default value) > Size (in bytes) of "phase 1" fragment sent for all > large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_openib_eager_limit" (current value: > "12288", data source: default value) > Maximum size (in bytes) of "short" messages (must > be >= 1). > MCA btl: parameter "btl_openib_max_send_size" (current > value: "65536", data source: default value) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_openib_rdma_pipeline_send_length" > (current value: "1048576", data source: default value) > Length of the "phase 2" portion of a large message > (in bytes) when using the pipeline protocol. This part of the message will > be split into fragments of size max_send_size and sent using send/receive > semantics (must be >= 0; only relevant when the PUT flag is set) > MCA btl: parameter "btl_openib_rdma_pipeline_frag_size" > (current value: "1048576", data source: default value) > Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant when > the PUT flag is set) > MCA btl: parameter "btl_openib_min_rdma_pipeline_size" > (current value: "262144", data source: default value) > Messages smaller than this size (in bytes) will not > use the RDMA pipeline protocol. Instead, they will be split into fragments > of max_send_size and sent using send/receive semantics (must be >=0, and is > automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag > is set) > MCA btl: parameter "btl_openib_bandwidth" (current value: > "800", data source: default value) > Approximate maximum bandwidth of interconnect(must > be >= 1) > MCA btl: parameter "btl_openib_latency" (current value: > "10", data source: default value) > Approximate latency of interconnect (must be >= 0) > MCA btl: parameter "btl_openib_receive_queues" (current > value: > "P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32", > data source: default value) > Colon-delimited, comma delimited list of receive > queues: P,4096,8,6,4:P,32768,8,6,4 > MCA btl: parameter "btl_openib_if_include" (current value: > <none>, data source: default value) > Comma-delimited list of devices/ports to be used > (e.g. "mthca0,mthca1:2"; empty value means to use all ports found). Mutually > exclusive with btl_openib_if_exclude. > MCA btl: parameter "btl_openib_if_exclude" (current value: > <none>, data source: default value) > Comma-delimited list of device/ports to be excluded > (empty value means to not exclude any ports). Mutually exclusive with > btl_openib_if_include. > MCA btl: parameter "btl_openib_ipaddr_include" (current > value: <none>, data source: default value) > Comma-delimited list of IP Addresses to be used > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude. > MCA btl: parameter "btl_openib_ipaddr_exclude" (current > value: <none>, data source: default value) > Comma-delimited list of IP Addresses to be excluded > (e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include. > MCA btl: parameter "btl_openib_cpc_include" (current value: > <none>, data source: default value) > Method used to select OpenFabrics connections > (valid values: oob,xoob,rdmacm) > MCA btl: parameter "btl_openib_cpc_exclude" (current value: > <none>, data source: default value) > Method used to exclude OpenFabrics connections > (valid values: oob,xoob,rdmacm) > MCA btl: parameter "btl_openib_connect_oob_priority" > (current value: "50", data source: default value) > The selection method priority for oob > MCA btl: parameter "btl_openib_connect_xoob_priority" > (current value: "60", data source: default value) > The selection method priority for xoob > MCA btl: parameter "btl_openib_connect_rdmacm_priority" > (current value: "30", data source: default value) > The selection method priority for rdma_cm > MCA btl: parameter "btl_openib_connect_rdmacm_port" (current > value: "0", data source: default value) > The selection method port for rdma_cm > MCA btl: parameter > "btl_openib_connect_rdmacm_resolve_timeout" (current value: "30000", data > source: default value) > The timeout (in miliseconds) for address and route > resolution > MCA btl: parameter "btl_openib_connect_rdmacm_retry_count" > (current value: "20", data source: default value) > Maximum number of times rdmacm will retry route > resolution > MCA btl: parameter > "btl_openib_connect_rdmacm_reject_causes_connect_error" (current value: "0", > data source: default value) > The drivers for some devices are buggy such that an > RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED > event. Setting this MCA parameter to true tells Open MPI to treat > CONNECT_ERROR events on connections where a REJECT is expected as a REJECT > (default: false) > MCA btl: parameter "btl_openib_priority" (current value: > "0", data source: default value) > MCA btl: parameter "btl_tcp_links" (current value: "1", data > source: default value) > MCA btl: parameter "btl_tcp_if_include" (current value: > <none>, data source: default value) > MCA btl: parameter "btl_tcp_if_exclude" (current value: > "lo", data source: default value) > MCA btl: parameter "btl_tcp_free_list_num" (current value: > "8", data source: default value) > MCA btl: parameter "btl_tcp_free_list_max" (current value: > "-1", data source: default value) > MCA btl: parameter "btl_tcp_free_list_inc" (current value: > "32", data source: default value) > MCA btl: parameter "btl_tcp_sndbuf" (current value: > "131072", data source: default value) > MCA btl: parameter "btl_tcp_rcvbuf" (current value: > "131072", data source: default value) > MCA btl: parameter "btl_tcp_endpoint_cache" (current value: > "30720", data source: default value) > The size of the internal cache for each TCP > connection. This cache is used to reduce the number of syscalls, by replacing > them with memcpy. Every read will read the expected data plus the amount of > the endpoint_cache > MCA btl: parameter "btl_tcp_use_nagle" (current value: "0", > data source: default value) > Whether to use Nagle's algorithm or not (using > Nagle's algorithm may increase short message latency) > MCA btl: parameter "btl_tcp_port_min_v4" (current value: > "1024", data source: default value) > The minimum port where the TCP BTL will try to bind > (default 1024) > MCA btl: parameter "btl_tcp_port_range_v4" (current value: > "64511", data source: default value) > The number of ports where the TCP BTL will try to > bind (default 64511). This parameter together with the port min, define a > range of ports where Open MPI will open sockets. > MCA btl: parameter "btl_tcp_port_min_v6" (current value: > "1024", data source: default value) > The minimum port where the TCP BTL will try to bind > (default 1024) > MCA btl: parameter "btl_tcp_port_range_v6" (current value: > "64511", data source: default value) > The number of ports where the TCP BTL will try to > bind (default 64511). This parameter together with the port min, define a > range of ports where Open MPI will open sockets. > MCA btl: parameter "btl_tcp_exclusivity" (current value: > "100", data source: default value) > BTL exclusivity (must be >= 0) > MCA btl: parameter "btl_tcp_flags" (current value: "314", > data source: default value) > BTL bit flags (general flags: SEND=1, PUT=2, GET=4, > SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by > the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128) > MCA btl: parameter "btl_tcp_rndv_eager_limit" (current > value: "65536", data source: default value) > Size (in bytes) of "phase 1" fragment sent for all > large messages (must be >= 0 and <= eager_limit) > MCA btl: parameter "btl_tcp_eager_limit" (current value: > "65536", data source: default value) > Maximum size (in bytes) of "short" messages (must > be >= 1). > MCA btl: parameter "btl_tcp_max_send_size" (current value: > "131072", data source: default value) > Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > MCA btl: parameter "btl_tcp_rdma_pipeline_send_length" > (current value: "131072", data source: default value) > Length of the "phase 2" portion of a large message > (in bytes) when using the pipeline protocol. This part of the message will > be split into fragments of size max_send_size and sent using send/receive > semantics (must be >= 0; only relevant when the PUT flag is set) > MCA btl: parameter "btl_tcp_rdma_pipeline_frag_size" > (current value: "2147483647", data source: default value) > Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant when > the PUT flag is set) > MCA btl: parameter "btl_tcp_min_rdma_pipeline_size" (current > value: "0", data source: default value) > Messages smaller than this size (in bytes) will not > use the RDMA pipeline protocol. Instead, they will be split into fragments > of max_send_size and sent using send/receive semantics (must be >=0, and is > automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag > is set) > MCA btl: parameter "btl_tcp_bandwidth" (current value: > "100", data source: default value) > Approximate maximum bandwidth of interconnect(must > be >= 1) > MCA btl: parameter "btl_tcp_latency" (current value: "100", > data source: default value) > Approximate latency of interconnect (must be >= 0) > MCA btl: parameter "btl_tcp_disable_family" (current value: > "0", data source: default value) > MCA btl: parameter "btl_tcp_priority" (current value: "0", > data source: default value) > MCA btl: parameter "btl_base_include" (current value: > <none>, data source: default value) > MCA btl: parameter "btl_base_exclude" (current value: > <none>, data source: default value) > MCA btl: parameter "btl_base_warn_component_unused" (current > value: "1", data source: default value) > This parameter is used to turn on warning messages > when certain NICs are not used > MCA mtl: parameter "mtl" (current value: <none>, data > source: default value) > Default selection set of components for the mtl > framework (<none> means use all components that can be found) > MCA mtl: parameter "mtl_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the mtl framework (0 = no > verbosity) > MCA topo: parameter "topo" (current value: <none>, data > source: default value) > Default selection set of components for the topo > framework (<none> means use all components that can be found) > MCA topo: parameter "topo_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the topo framework (0 = no > verbosity) > MCA topo: parameter "topo_unity_priority" (current value: > "0", data source: default value) > MCA osc: parameter "osc" (current value: <none>, data > source: default value) > Default selection set of components for the osc > framework (<none> means use all components that can be found) > MCA osc: parameter "osc_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the osc framework (0 = no > verbosity) > MCA osc: parameter "osc_pt2pt_no_locks" (current value: "0", > data source: default value) > Enable optimizations available only if MPI_LOCK is > not used. > MCA osc: parameter "osc_pt2pt_eager_limit" (current value: > "16384", data source: default value) > Max size of eagerly sent data > MCA osc: parameter "osc_pt2pt_priority" (current value: "0", > data source: default value) > MCA osc: parameter "osc_rdma_eager_send" (current value: > "1", data source: default value) > Attempt to start data movement during communication > call, instead of at synchrnoization time. Info key of same name overrides > this value. > MCA osc: parameter "osc_rdma_use_buffers" (current value: > "0", data source: default value) > Coalesce messages during an epoch to reduce network > utilization. Info key of same name overrides this value. > MCA osc: parameter "osc_rdma_use_rdma" (current value: "0", > data source: default value) > Use real RDMA operations to transfer data. Info > key of same name overrides this value. > MCA osc: parameter "osc_rdma_rdma_completion_wait" (current > value: "1", data source: default value) > Wait for all completion of rdma events before > sending acknowledgment. Info key of same name overrides this value. > MCA osc: parameter "osc_rdma_no_locks" (current value: "0", > data source: default value) > Enable optimizations available only if MPI_LOCK is > not used. Info key of same name overrides this value. > MCA osc: parameter "osc_rdma_priority" (current value: "0", > data source: default value) > MCA iof: parameter "iof" (current value: <none>, data > source: default value) > Default selection set of components for the iof > framework (<none> means use all components that can be found) > MCA iof: parameter "iof_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the iof framework (0 = no > verbosity) > MCA iof: parameter "iof_hnp_priority" (current value: "0", > data source: default value) > MCA iof: parameter "iof_orted_priority" (current value: "0", > data source: default value) > MCA iof: parameter "iof_tool_priority" (current value: "0", > data source: default value) > MCA oob: parameter "oob" (current value: <none>, data > source: default value) > Default selection set of components for the oob > framework (<none> means use all components that can be found) > MCA oob: parameter "oob_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the oob framework (0 = no > verbosity) > MCA oob: parameter "oob_tcp_verbose" (current value: "0", > data source: default value) > Verbose level for the OOB tcp component > MCA oob: parameter "oob_tcp_peer_limit" (current value: > "-1", data source: default value) > Maximum number of peer connections to > simultaneously maintain (-1 = infinite) > MCA oob: parameter "oob_tcp_peer_retries" (current value: > "60", data source: default value) > Number of times to try shutting down a connection > before giving up > MCA oob: parameter "oob_tcp_debug" (current value: "0", data > source: default value) > Enable (1) / disable (0) debugging output for this > component > MCA oob: parameter "oob_tcp_sndbuf" (current value: > "131072", data source: default value) > TCP socket send buffering size (in bytes) > MCA oob: parameter "oob_tcp_rcvbuf" (current value: > "131072", data source: default value) > TCP socket receive buffering size (in bytes) > MCA oob: parameter "oob_tcp_if_include" (current value: > <none>, data source: default value) > Comma-delimited list of TCP interfaces to use > MCA oob: parameter "oob_tcp_if_exclude" (current value: > <none>, data source: default value) > Comma-delimited list of TCP interfaces to exclude > MCA oob: parameter "oob_tcp_connect_sleep" (current value: > "1", data source: default value) > Enable (1) / disable (0) random sleep for > connection wireup. > MCA oob: parameter "oob_tcp_listen_mode" (current value: > "event", data source: default value) > Mode for HNP to accept incoming connections: event, > listen_thread. > MCA oob: parameter "oob_tcp_listen_thread_max_queue" > (current value: "10", data source: default value) > High water mark for queued accepted socket list > size. Used only when listen_mode is listen_thread. > MCA oob: parameter "oob_tcp_listen_thread_wait_time" > (current value: "10", data source: default value) > Time in milliseconds to wait before actively > checking for new connections when listen_mode is listen_thread. > MCA oob: parameter "oob_tcp_port_min_v4" (current value: > "0", data source: default value) > Starting port allowed (IPv4) > MCA oob: parameter "oob_tcp_port_range_v4" (current value: > "65535", data source: default value) > Range of allowed ports (IPv4) > MCA oob: parameter "oob_tcp_disable_family" (current value: > "0", data source: default value) > Disable IPv4 (4) or IPv6 (6) > MCA oob: parameter "oob_tcp_port_min_v6" (current value: > "0", data source: default value) > Starting port allowed (IPv6) > MCA oob: parameter "oob_tcp_port_range_v6" (current value: > "65535", data source: default value) > Range of allowed ports (IPv6) > MCA oob: parameter "oob_tcp_priority" (current value: "0", > data source: default value) > MCA odls: parameter "odls_base_sigkill_timeout" (current > value: "1", data source: default value) > Time to wait for a process to die after issuing a > kill signal to it > MCA odls: parameter "odls_base_report_bindings" (current > value: "0", data source: default value) > Report process bindings [default: no] > MCA odls: parameter "odls" (current value: <none>, data > source: default value) > Default selection set of components for the odls > framework (<none> means use all components that can be found) > MCA odls: parameter "odls_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the odls framework (0 = no > verbosity) > MCA odls: parameter "odls_default_priority" (current value: > "0", data source: default value) > MCA ras: parameter "ras_base_display_alloc" (current value: > "0", data source: default value) > Whether to display the allocation after it is > determined > MCA ras: parameter "ras_base_display_devel_alloc" (current > value: "0", data source: default value) > Whether to display a developer-detail allocation > after it is determined > MCA ras: parameter "ras" (current value: <none>, data > source: default value) > Default selection set of components for the ras > framework (<none> means use all components that can be found) > MCA ras: parameter "ras_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the ras framework (0 = no > verbosity) > MCA ras: parameter "ras_slurm_priority" (current value: > "75", data source: default value) > Priority of the slurm ras component > MCA ras: parameter "ras_tm_priority" (current value: "100", > data source: default value) > Priority of the tm ras component > MCA ras: parameter "ras_tm_nodefile_dir" (current value: > "/var/spool/torque/aux", data source: default value) > The directory where the PBS nodefile can be found > MCA rmaps: parameter "rmaps_rank_file_path" (current value: > <none>, data source: default value, synonym of: orte_rankfile) > Name of the rankfile to be used for mapping > processes (relative or absolute path) > MCA rmaps: parameter "rmaps_base_schedule_policy" (current > value: "slot", data source: default value) > Scheduling Policy for RMAPS. [slot (alias:core) | > socket | board | node] > MCA rmaps: parameter "rmaps_base_pernode" (current value: "0", > data source: default value) > Launch one ppn as directed > MCA rmaps: parameter "rmaps_base_n_pernode" (current value: > "-1", data source: default value) > Launch n procs/node > MCA rmaps: parameter "rmaps_base_n_perboard" (current value: > "-1", data source: default value) > Launch n procs/board > MCA rmaps: parameter "rmaps_base_n_persocket" (current value: > "-1", data source: default value) > Launch n procs/socket > MCA rmaps: parameter "rmaps_base_loadbalance" (current value: > "0", data source: default value) > Balance total number of procs across all allocated > nodes > MCA rmaps: parameter "rmaps_base_cpus_per_proc" (current > value: "1", data source: default value, synonyms: rmaps_base_cpus_per_rank) > Number of cpus to use for each rank [1-2**15 > (default=1)] > MCA rmaps: parameter "rmaps_base_cpus_per_rank" (current > value: "1", data source: default value, synonym of: rmaps_base_cpus_per_proc) > Number of cpus to use for each rank [1-2**15 > (default=1)] > MCA rmaps: parameter "rmaps_base_stride" (current value: "1", > data source: default value) > When binding multiple cores to a rank, the step > size to use between cores [1-2**15 (default: 1)] > MCA rmaps: parameter "rmaps_base_slot_list" (current value: > <none>, data source: default value) > List of processor IDs to bind MPI processes to > (e.g., used in conjunction with rank files) [default=NULL] > MCA rmaps: parameter "rmaps_base_no_schedule_local" (current > value: "0", data source: default value) > If false, allow scheduling MPI applications on the > same node as mpirun (default). If true, do not schedule any MPI applications > on the same node as mpirun > MCA rmaps: parameter "rmaps_base_no_oversubscribe" (current > value: "0", data source: default value) > If true, then do not allow oversubscription of > nodes - mpirun will return an error if there aren't enough nodes to launch > all processes without oversubscribing > MCA rmaps: parameter "rmaps_base_display_map" (current value: > "0", data source: default value) > Whether to display the process map after it is > computed > MCA rmaps: parameter "rmaps_base_display_devel_map" (current > value: "0", data source: default value) > Whether to display a developer-detail process map > after it is computed > MCA rmaps: parameter "rmaps" (current value: <none>, data > source: default value) > Default selection set of components for the rmaps > framework (<none> means use all components that can be found) > MCA rmaps: parameter "rmaps_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the rmaps framework (0 = no > verbosity) > MCA rmaps: parameter "rmaps_load_balance_priority" (current > value: "0", data source: default value) > MCA rmaps: parameter "rmaps_rank_file_priority" (current > value: "0", data source: default value) > MCA rmaps: parameter "rmaps_round_robin_priority" (current > value: "0", data source: default value) > MCA rmaps: parameter "rmaps_seq_priority" (current value: "0", > data source: default value) > MCA rml: parameter "rml_wrapper" (current value: <none>, > data source: default value) > Use a Wrapper component around the selected RML > component > MCA rml: parameter "rml" (current value: <none>, data > source: default value) > Default selection set of components for the rml > framework (<none> means use all components that can be found) > MCA rml: parameter "rml_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the rml framework (0 = no > verbosity) > MCA rml: parameter "rml_oob_priority" (current value: "0", > data source: default value) > MCA routed: parameter "routed" (current value: <none>, data > source: default value) > Default selection set of components for the routed > framework (<none> means use all components that can be found) > MCA routed: parameter "routed_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the routed framework (0 = no > verbosity) > MCA routed: parameter "routed_binomial_priority" (current > value: "0", data source: default value) > MCA routed: parameter "routed_direct_priority" (current value: > "0", data source: default value) > MCA routed: parameter "routed_linear_priority" (current value: > "0", data source: default value) > MCA plm: parameter "plm" (current value: <none>, data > source: default value) > Default selection set of components for the plm > framework (<none> means use all components that can be found) > MCA plm: parameter "plm_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the plm framework (0 = no > verbosity) > MCA plm: parameter "plm_rsh_num_concurrent" (current value: > "128", data source: default value) > How many plm_rsh_agent instances to invoke > concurrently (must be > 0) > MCA plm: parameter "plm_rsh_force_rsh" (current value: "0", > data source: default value) > Force the launcher to always use rsh > MCA plm: parameter "plm_rsh_disable_qrsh" (current value: > "0", data source: default value) > Disable the launcher to use qrsh when under the SGE > parallel environment > MCA plm: parameter "plm_rsh_daemonize_qrsh" (current value: > "0", data source: default value) > Daemonize the orted under the SGE parallel > environment > MCA plm: parameter "plm_rsh_priority" (current value: "10", > data source: default value) > Priority of the rsh plm component > MCA plm: parameter "plm_rsh_delay" (current value: "1", data > source: default value) > Delay (in seconds) between invocations of the > remote agent, but only used when the "debug" MCA parameter is true, or the > top-level MCA debugging is enabled (otherwise this value is ignored) > MCA plm: parameter "plm_rsh_assume_same_shell" (current > value: "1", data source: default value) > If set to 1, assume that the shell on the remote > node is the same as the shell on the local node. Otherwise, probe for what > the remote shell. > MCA plm: parameter "plm_rsh_agent" (current value: "ssh : > rsh", data source: default value, synonyms: pls_rsh_agent) > The command used to launch executables on remote > nodes (typically either "ssh" or "rsh") > MCA plm: parameter "plm_rsh_tree_spawn" (current value: "0", > data source: default value) > If set to 1, launch via a tree-based topology > MCA plm: parameter "plm_slurm_args" (current value: <none>, > data source: default value) > Custom arguments to srun > MCA plm: parameter "plm_slurm_priority" (current value: "0", > data source: default value) > MCA plm: parameter "plm_tm_want_path_check" (current value: > "1", data source: default value) > Whether the launching process should check for the > plm_tm_orted executable in the PATH before launching (the TM API does not > give an indication of failure; this is a somewhat-lame workaround; non-zero > values enable this check) > MCA plm: parameter "plm_tm_priority" (current value: "0", > data source: default value) > MCA filem: parameter "filem" (current value: <none>, data > source: default value) > Which Filem component to use (empty = auto-select) > MCA filem: parameter "filem_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the filem framework (0 = no > verbosity) > MCA filem: parameter "filem_rsh_priority" (current value: > "20", data source: default value) > Priority of the FILEM rsh component > MCA filem: parameter "filem_rsh_verbose" (current value: "0", > data source: default value) > Verbose level for the FILEM rsh component > MCA filem: parameter "filem_rsh_rcp" (current value: "scp", > data source: default value) > The rsh cp command for the FILEM rsh component > MCA filem: parameter "filem_rsh_rsh" (current value: "ssh", > data source: default value) > The remote shell command for the FILEM rsh component > MCA filem: parameter "filem_rsh_max_incomming" (current value: > "10", data source: default value) > Maximum number of incomming connections > MCA filem: parameter "filem_rsh_max_outgoing" (current value: > "10", data source: default value) > Maximum number of out going connections (Currently > not used) > MCA errmgr: parameter "errmgr" (current value: <none>, data > source: default value) > Default selection set of components for the errmgr > framework (<none> means use all components that can be found) > MCA errmgr: parameter "errmgr_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the errmgr framework (0 = no > verbosity) > MCA errmgr: parameter "errmgr_default_priority" (current value: > "0", data source: default value) > MCA ess: parameter "ess" (current value: <none>, data > source: default value) > Default selection set of components for the ess > framework (<none> means use all components that can be found) > MCA ess: parameter "ess_base_verbose" (current value: "0", > data source: default value) > Verbosity level for the ess framework (0 = no > verbosity) > MCA ess: parameter "ess_env_priority" (current value: "0", > data source: default value) > MCA ess: parameter "ess_hnp_priority" (current value: "0", > data source: default value) > MCA ess: parameter "ess_singleton_priority" (current value: > "0", data source: default value) > MCA ess: parameter "ess_slurm_priority" (current value: "0", > data source: default value) > MCA ess: parameter "ess_tool_priority" (current value: "0", > data source: default value) > MCA grpcomm: parameter "grpcomm" (current value: <none>, data > source: default value) > Default selection set of components for the grpcomm > framework (<none> means use all components that can be found) > MCA grpcomm: parameter "grpcomm_base_verbose" (current value: > "0", data source: default value) > Verbosity level for the grpcomm framework (0 = no > verbosity) > MCA grpcomm: parameter "grpcomm_bad_priority" (current value: > "0", data source: default value) > MCA grpcomm: parameter "grpcomm_basic_priority" (current value: > "0", data source: default value) > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA