Dear Open MPI developer,We're puzzled by reproducible performance (bandwidth) penalty observed when comparing measurements via InfibiBand between two nodes, OpenMPI/1.10.0 compiled with *GCC/5.2* instead of GCC 4.8 and Intel compiler.
Take a look at the attached picture of two measurements of NetPIPE http://bitspjoule.org/netpipe/ benchmark done with one MPI rank per node, communicating via QDR InfiniBand (y axis: Mbps, y axis: sample number)
Up to sample 64 (8195 bytes message size) the achieved performance is virtually the same; from sample 65 (12285 bytes, *less* than 12k) the version of GCC compiled using GCC 5.2 suffer form 20%+ penalty in bandwidth.
The result is reproducible and independent from nodes and ever linux distribution (both Scientific Linux 6 and CentOS 7 have the same results). Both C and Fortran benchmarks offer the very same behaviour so it is *not* an f08 issue.
The acchieved bandwidth is definitely IB-range (gigabytes per second), the communication is running via InfinfiBand in all cases (no failback to IP, huh).
The compile line is the same; the output of ompi_info --all and --params is the very same (cf. attachments) up to added support for fortran-08 in /5.2 version.
We know about existence of 'eager_limit' parameter, which is *not* changed and is 12288 in both versions (this is *less* that the first distinguishing sample).
Again, for us the *only* difference is usage of other (new) GCC release. Any idea about this 20%+ bandwidth loss? Best Paul Kapinos -- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, IT Center Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
MCA btl: parameter "btl_openib_verbose" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
Output some verbose OpenIB BTL information (0 = no
output, nonzero = output)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_no_device_params_found"
(current value: "true", data source: default, level: 9 dev/all, type: bool,
synonyms: btl_openib_warn_no_hca_params_found)
Warn when no device-specific parameters are found in
the INI file specified by the btl_openib_device_param_files MCA parameter (0 =
do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_no_hca_params_found"
(current value: "true", data source: default, level: 9 dev/all, type: bool,
deprecated, synonym of: btl_openib_warn_no_device_params_found)
Warn when no device-specific parameters are found in
the INI file specified by the btl_openib_device_param_files MCA parameter (0 =
do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_default_gid_prefix"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
Warn when there is more than one active ports and at
least one of them connected to the network with only default GID prefix
configured (0 = do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_nonexistent_if" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Warn if non-existent devices and/or ports are
specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn;
any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_abort_not_enough_reg_mem"
(current value: "false", data source: default, level: 9 dev/all, type: bool)
If there is not enough registered memory available on
the system for Open MPI to function properly, Open MPI will issue a warning.
If this MCA parameter is set to true, then Open MPI will also abort all MPI
jobs (0 = warn, but do not abort; any other value = warn and abort)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_poll_cq_batch" (current value:
"256", data source: default, level: 9 dev/all, type: unsigned)
Retrieve up to poll_cq_batch completions from CQ
MCA btl: parameter "btl_openib_device_param_files" (current
value:
"/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini",
data source: default, level: 9 dev/all, type: string, synonyms:
btl_openib_hca_param_files)
Colon-delimited list of INI-style files that contain
device vendor/part-specific parameters (use semicolon for Windows)
MCA btl: parameter "btl_openib_hca_param_files" (current
value:
"/opt/MPI/openmpi-1.10.0/linux/gcc/share/openmpi/mca-btl-openib-device-params.ini",
data source: default, level: 9 dev/all, type: string, deprecated, synonym of:
btl_openib_device_param_files)
Colon-delimited list of INI-style files that contain
device vendor/part-specific parameters (use semicolon for Windows)
MCA btl: parameter "btl_openib_device_type" (current value:
"all", data source: default, level: 9 dev/all, type: int)
Specify to only use IB or iWARP network adapters
(infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = use
any available adapters)
Valid values: 0:"infiniband", 0:"ib", 1:"iwarp",
1:"iw", 2:"all"
MCA btl: parameter "btl_openib_max_btls" (current value: "-1",
data source: default, level: 9 dev/all, type: int)
Maximum number of device ports to use (-1 = use all
available, otherwise must be >= 1)
MCA btl: parameter "btl_openib_free_list_num" (current value:
"8", data source: default, level: 9 dev/all, type: int)
Initial size of free lists (must be >= 1)
MCA btl: parameter "btl_openib_free_list_max" (current value:
"-1", data source: default, level: 9 dev/all, type: int)
Maximum size of free lists (-1 = infinite, otherwise
must be >= 0)
MCA btl: parameter "btl_openib_free_list_inc" (current value:
"32", data source: default, level: 9 dev/all, type: int)
Increment size of free lists (must be >= 1)
MCA btl: parameter "btl_openib_mpool" (current value: "grdma",
data source: default, level: 9 dev/all, type: string)
Name of the memory pool to be used (it is unlikely
that you will ever want to change this)
MCA btl: parameter "btl_openib_reg_mru_len" (current value:
"16", data source: default, level: 9 dev/all, type: int)
Length of the registration cache most recently used
list (must be >= 1)
MCA btl: parameter "btl_openib_cq_size" (current value:
"8192", data source: default, level: 9 dev/all, type: int, synonyms:
btl_openib_ib_cq_size)
Minimum size of the OpenFabrics completion queue (CQs
are automatically sized based on the number of peer MPI processes; this value
determines the *minimum* size of all CQs)
MCA btl: parameter "btl_openib_ib_cq_size" (current value:
"8192", data source: default, level: 9 dev/all, type: int, deprecated, synonym
of: btl_openib_cq_size)
Minimum size of the OpenFabrics completion queue (CQs
are automatically sized based on the number of peer MPI processes; this value
determines the *minimum* size of all CQs)
MCA btl: parameter "btl_openib_max_inline_data" (current
value: "-1", data source: default, level: 9 dev/all, type: int, synonyms:
btl_openib_ib_max_inline_data)
Maximum size of inline data segment (-1 = run-time
probe to discover max value, otherwise must be >= 0). If not explicitly set,
use max_inline_data from the INI file containing device-specific parameters
MCA btl: parameter "btl_openib_ib_max_inline_data" (current
value: "-1", data source: default, level: 9 dev/all, type: int, deprecated,
synonym of: btl_openib_max_inline_data)
Maximum size of inline data segment (-1 = run-time
probe to discover max value, otherwise must be >= 0). If not explicitly set,
use max_inline_data from the INI file containing device-specific parameters
MCA btl: parameter "btl_openib_pkey" (current value: "0", data
source: default, level: 9 dev/all, type: unsigned, synonyms:
btl_openib_ib_pkey_val)
OpenFabrics partition key (pkey) value. Unsigned
integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be
masked against the maximum allowable IB partition key value (0x7fff)
MCA btl: parameter "btl_openib_ib_pkey_val" (current value:
"0", data source: default, level: 9 dev/all, type: unsigned, deprecated,
synonym of: btl_openib_pkey)
OpenFabrics partition key (pkey) value. Unsigned
integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be
masked against the maximum allowable IB partition key value (0x7fff)
MCA btl: parameter "btl_openib_psn" (current value: "0", data
source: default, level: 9 dev/all, type: unsigned, synonyms: btl_openib_ib_psn)
OpenFabrics packet sequence starting number (must be
>= 0)
MCA btl: parameter "btl_openib_ib_psn" (current value: "0",
data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym of:
btl_openib_psn)
OpenFabrics packet sequence starting number (must be
>= 0)
MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current
value: "4", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand outstanding atomic reads (must be >= 0)
MCA btl: parameter "btl_openib_mtu" (current value: "1k", data
source: default, level: 9 dev/all, type: int, synonyms: btl_openib_ib_mtu)
OpenFabrics MTU, in bytes (if not specified in INI
files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048
bytes, 5=4096 bytes
Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k",
5:"4k"
MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k",
data source: default, level: 9 dev/all, type: int, deprecated, synonym of:
btl_openib_mtu)
OpenFabrics MTU, in bytes (if not specified in INI
files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048
bytes, 5=4096 bytes
Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k",
5:"4k"
MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current
value: "25", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand minimum "receiver not ready" timer, in
seconds (must be >= 0 and <= 31)
MCA btl: parameter "btl_openib_ib_timeout" (current value:
"24", data source: environment, level: 9 dev/all, type: unsigned)
InfiniBand transmit timeout, plugged into formula:
4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31)
MCA btl: parameter "btl_openib_ib_retry_count" (current value:
"7", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand transmit retry count (must be >= 0 and <=
7)
MCA btl: parameter "btl_openib_ib_rnr_retry" (current value:
"7", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand "receiver not ready" retry count; applies
*only* to SRQ/XRC queues. PP queues use RNR retry values of 0 because Open MPI
performs software flow control to guarantee that RNRs never occur (must be >= 0
and <= 7; 7 = "infinite")
MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current
value: "4", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand maximum pending RDMA destination
operations (must be >= 0)
MCA btl: parameter "btl_openib_ib_service_level" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand service level (must be >= 0 and <= 15)
MCA btl: parameter "btl_openib_ib_path_record_service_level"
(current value: "0", data source: default, level: 9 dev/all, type: unsigned)
Enable getting InfiniBand service level from
PathRecord (must be >= 0, 0 = disabled, positive = try to get the service level
from PathRecord)
MCA btl: parameter "btl_openib_use_eager_rdma" (current value:
"-1", data source: default, level: 9 dev/all, type: int)
Use RDMA for eager messages (-1 = use device default,
0 = do not use eager RDMA, 1 = use eager RDMA)
MCA btl: parameter "btl_openib_eager_rdma_threshold" (current
value: "16", data source: default, level: 9 dev/all, type: int)
Use RDMA for short messages after this number of
messages are received from a given peer (must be >= 1)
MCA btl: parameter "btl_openib_max_eager_rdma" (current value:
"16", data source: default, level: 9 dev/all, type: int)
Maximum number of peers allowed to use RDMA for short
messages (RDMA is used for all long messages, except if explicitly disabled,
such as with the "dr" pml) (must be >= 0)
MCA btl: parameter "btl_openib_eager_rdma_num" (current value:
"17", data source: default, level: 9 dev/all, type: int)
Number of RDMA buffers to allocate for small messages
(must be >= 1)
MCA btl: parameter "btl_openib_btls_per_lid" (current value:
"1", data source: default, level: 9 dev/all, type: unsigned)
Number of BTLs to create for each InfiniBand LID
(must be >= 1)
MCA btl: parameter "btl_openib_max_lmc" (current value: "1",
data source: default, level: 9 dev/all, type: unsigned)
Maximum number of LIDs to use for each device port
(must be >= 0, where 0 = use all available)
MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Maximum number of alternative paths for each device
port (must be >= -1, where 0 = disable apm, -1 = all available alternative
paths )
MCA btl: parameter "btl_openib_enable_apm_over_ports" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Enable alternative path migration (APM) over
different ports of the same device (must be >= 0, where 0 = disable APM over
ports, 1 = enable APM over ports of the same device)
MCA btl: parameter "btl_openib_use_async_event_thread"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
If nonzero, use the thread that will handle
InfiniBand asynchronous events
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_enable_srq_resize" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Enable/Disable on demand SRQ resize. (0 = without
resizing, nonzero = with resizing)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_rroce_enable" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
Enable/Disable routing between different subnets(0 =
disable, nonzero = enable)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_buffer_alignment" (current
value: "64", data source: default, level: 9 dev/all, type: unsigned)
Preferred communication buffer alignment, in bytes
(must be > 0 and power of two)
MCA btl: parameter "btl_openib_cq_poll_ratio" (current value:
"100", data source: default, level: 9 dev/all, type: unsigned)
How often to poll high priority CQ versus low
priority CQ
MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" (current
value: "100", data source: default, level: 9 dev/all, type: unsigned)
How often to poll eager RDMA channel versus CQ
MCA btl: parameter "btl_openib_hp_cq_poll_per_progress"
(current value: "10", data source: default, level: 9 dev/all, type: unsigned)
Max number of completion events to process for each
call of BTL progress engine
MCA btl: parameter "btl_openib_max_hw_msg_size" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
Maximum size (in bytes) of a single fragment of a
long message when using the RDMA protocols (must be > 0 and <= hw capabilities).
MCA btl: parameter "btl_openib_allow_max_memory_registration"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
Allow maximum possible memory to register with HCA
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_memory_registration_verbose"
(current value: "0", data source: default, level: 9 dev/all, type: int)
Output some verbose memory registration information
(0 = no output, nonzero = output)
MCA btl: parameter "btl_openib_ignore_locality" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Ignore any locality information and use all devices
(0 = use locality informaiton and use only close devices, nonzero = ignore
locality information)
MCA btl: informational "btl_openib_have_fork_support" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Whether the OpenFabrics stack supports applications
that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that this
value does NOT indicate whether the system being run on supports "fork()" with
OpenFabrics applications or not.
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_exclusivity" (current value:
"1024", data source: default, level: 7 dev/basic, type: unsigned)
BTL exclusivity (must be >= 0)
MCA btl: parameter "btl_openib_flags" (current value: "310",
data source: default, level: 5 tuner/detail, type: unsigned)
BTL bit flags (general flags: SEND=1, PUT=2, GET=4,
SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by the
"dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; flags
only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512)
MCA btl: parameter "btl_openib_rndv_eager_limit" (current
value: "12288", data source: default, level: 4 tuner/basic, type: size_t)
Size (in bytes, including header) of "phase 1"
fragment sent for all large messages (must be >= 0 and <= eager_limit)
MCA btl: parameter "btl_openib_eager_limit" (current value:
"12288", data source: default, level: 4 tuner/basic, type: size_t)
Maximum size (in bytes, including header) of "short"
messages (must be >= 1).
MCA btl: parameter "btl_openib_max_send_size" (current value:
"65536", data source: default, level: 4 tuner/basic, type: size_t)
Maximum size (in bytes) of a single "phase 2"
fragment of a long message when using the pipeline protocol (must be >= 1)
MCA btl: parameter "btl_openib_rdma_pipeline_send_length"
(current value: "1048576", data source: default, level: 4 tuner/basic, type:
size_t)
Length of the "phase 2" portion of a large message
(in bytes) when using the pipeline protocol. This part of the message will be
split into fragments of size max_send_size and sent using send/receive
semantics (must be >= 0; only relevant when the PUT flag is set)
MCA btl: parameter "btl_openib_rdma_pipeline_frag_size"
(current value: "1048576", data source: default, level: 4 tuner/basic, type:
size_t)
Maximum size (in bytes) of a single "phase 3"
fragment from a long message when using the pipeline protocol. These fragments
will be sent using RDMA semantics (must be >= 1; only relevant when the PUT
flag is set)
MCA btl: parameter "btl_openib_min_rdma_pipeline_size"
(current value: "1060864", data source: default, level: 4 tuner/basic, type:
size_t)
Messages smaller than this size (in bytes) will not
use the RDMA pipeline protocol. Instead, they will be split into fragments of
max_send_size and sent using send/receive semantics (must be >=0, and is
automatically adjusted up to at least
(eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag is
set)
MCA btl: parameter "btl_openib_latency" (current value: "4",
data source: default, level: 5 tuner/detail, type: unsigned)
Approximate latency of interconnect (0 = auto-detect
value at run-time [not supported in all BTL modules], >= 1 = latency in
microseconds)
MCA btl: parameter "btl_openib_bandwidth" (current value: "0",
data source: default, level: 5 tuner/detail, type: unsigned)
Approximate maximum bandwidth of interconnect (0 =
auto-detect value at run-time [not supported in all BTL modules], >= 1 =
bandwidth in Mbps)
MCA btl: parameter "btl_openib_receive_queues" (current value:
"P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64",
data source: default, level: 9 dev/all, type: string)
Colon-delimited, comma-delimited list of receive
queues: P,4096,8,6,4:P,32768,8,6,4
MCA btl: parameter "btl_openib_if_include" (current value: "",
data source: default, level: 9 dev/all, type: string)
Comma-delimited list of devices/ports to be used
(e.g. "mthca0,mthca1:2"; empty value means to use all ports found). Mutually
exclusive with btl_openib_if_exclude.
MCA btl: parameter "btl_openib_if_exclude" (current value: "",
data source: default, level: 9 dev/all, type: string)
Comma-delimited list of device/ports to be excluded
(empty value means to not exclude any ports). Mutually exclusive with
btl_openib_if_include.
MCA btl: parameter "btl_openib_ipaddr_include" (current value:
"", data source: default, level: 9 dev/all, type: string)
Comma-delimited list of IP Addresses to be used (e.g.
"192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude.
MCA btl: parameter "btl_openib_ipaddr_exclude" (current value:
"", data source: default, level: 9 dev/all, type: string)
Comma-delimited list of IP Addresses to be excluded
(e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include.
MCA btl: parameter "btl_openib_gid_index" (current value: "0",
data source: default, level: 9 dev/all, type: int)
GID index to use on verbs device ports
MCA btl: parameter "btl_openib_memalign" (current value: "32",
data source: default, level: 9 dev/all, type: int)
[64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0)
memoryalignment for all malloc calls if btl openib is used.
MCA btl: parameter "btl_openib_memalign_threshold" (current
value: "12288", data source: default, level: 9 dev/all, type: size_t)
Allocating memory more than
btl_openib_memalign_threshholdbytes will automatically be aligned to the value
of btl_openib_memalign bytes.memalign_threshhold defaults to the same value as
mca_btl_openib_eager_limit.
MCA btl: parameter "btl_openib_cpc_include" (current value:
"", data source: default, level: 9 dev/all, type: string)
Method used to select OpenFabrics connections (valid
values: rdmacm,udcm)
MCA btl: parameter "btl_openib_cpc_exclude" (current value:
"", data source: default, level: 9 dev/all, type: string)
Method used to exclude OpenFabrics connections (valid
values: rdmacm,udcm)
MCA btl: parameter "btl_openib_connect_rdmacm_priority"
(current value: "30", data source: default, level: 9 dev/all, type: int)
The selection method priority for rdma_cm
MCA btl: parameter "btl_openib_connect_rdmacm_port" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
The selection method port for rdma_cm
MCA btl: parameter "btl_openib_connect_rdmacm_resolve_timeout"
(current value: "30000", data source: default, level: 9 dev/all, type: int)
The timeout (in miliseconds) for address and route
resolution
MCA btl: parameter "btl_openib_connect_rdmacm_retry_count"
(current value: "20", data source: default, level: 9 dev/all, type: int)
Maximum number of times rdmacm will retry route
resolution
MCA btl: parameter
"btl_openib_connect_rdmacm_reject_causes_connect_error" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
The drivers for some devices are buggy such that an
RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED
event. Setting this MCA parameter to true tells Open MPI to treat
CONNECT_ERROR events on connections where a REJECT is expected as a REJECT
(default: false)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_connect_udcm_priority" (current
value: "63", data source: default, level: 9 dev/all, type: int)
Priority of the udcm connection method
MCA btl: parameter "btl_openib_connect_udcm_recv_count"
(current value: "512", data source: default, level: 9 dev/all, type: int)
Number of registered buffers to post
MCA btl: parameter "btl_openib_connect_udcm_timeout" (current
value: "500000", data source: default, level: 9 dev/all, type: int)
Ack timeout for udcm connection messages
MCA btl: parameter "btl_openib_connect_udcm_max_retry"
(current value: "25", data source: default, level: 9 dev/all, type: int)
Maximum number of times to retry sending a udcm
connection message
MCA btl: parameter "btl_openib_verbose" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
Output some verbose OpenIB BTL information (0 = no
output, nonzero = output)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_no_device_params_found"
(current value: "true", data source: default, level: 9 dev/all, type: bool,
synonyms: btl_openib_warn_no_hca_params_found)
Warn when no device-specific parameters are found in
the INI file specified by the btl_openib_device_param_files MCA parameter (0 =
do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_no_hca_params_found"
(current value: "true", data source: default, level: 9 dev/all, type: bool,
deprecated, synonym of: btl_openib_warn_no_device_params_found)
Warn when no device-specific parameters are found in
the INI file specified by the btl_openib_device_param_files MCA parameter (0 =
do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_default_gid_prefix"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
Warn when there is more than one active ports and at
least one of them connected to the network with only default GID prefix
configured (0 = do not warn; any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_warn_nonexistent_if" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Warn if non-existent devices and/or ports are
specified in the btl_openib_if_[in|ex]clude MCA parameters (0 = do not warn;
any other value = warn)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_abort_not_enough_reg_mem"
(current value: "false", data source: default, level: 9 dev/all, type: bool)
If there is not enough registered memory available on
the system for Open MPI to function properly, Open MPI will issue a warning.
If this MCA parameter is set to true, then Open MPI will also abort all MPI
jobs (0 = warn, but do not abort; any other value = warn and abort)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_poll_cq_batch" (current value:
"256", data source: default, level: 9 dev/all, type: unsigned)
Retrieve up to poll_cq_batch completions from CQ
MCA btl: parameter "btl_openib_device_param_files" (current
value:
"/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini",
data source: default, level: 9 dev/all, type: string, synonyms:
btl_openib_hca_param_files)
Colon-delimited list of INI-style files that contain
device vendor/part-specific parameters (use semicolon for Windows)
MCA btl: parameter "btl_openib_hca_param_files" (current
value:
"/opt/MPI/openmpi-1.10.0/linux/gcc_5.2.0/share/openmpi/mca-btl-openib-device-params.ini",
data source: default, level: 9 dev/all, type: string, deprecated, synonym of:
btl_openib_device_param_files)
Colon-delimited list of INI-style files that contain
device vendor/part-specific parameters (use semicolon for Windows)
MCA btl: parameter "btl_openib_device_type" (current value:
"all", data source: default, level: 9 dev/all, type: int)
Specify to only use IB or iWARP network adapters
(infiniband = only use InfiniBand HCAs; iwarp = only use iWARP NICs; all = use
any available adapters)
Valid values: 0:"infiniband", 0:"ib", 1:"iwarp",
1:"iw", 2:"all"
MCA btl: parameter "btl_openib_max_btls" (current value: "-1",
data source: default, level: 9 dev/all, type: int)
Maximum number of device ports to use (-1 = use all
available, otherwise must be >= 1)
MCA btl: parameter "btl_openib_free_list_num" (current value:
"8", data source: default, level: 9 dev/all, type: int)
Initial size of free lists (must be >= 1)
MCA btl: parameter "btl_openib_free_list_max" (current value:
"-1", data source: default, level: 9 dev/all, type: int)
Maximum size of free lists (-1 = infinite, otherwise
must be >= 0)
MCA btl: parameter "btl_openib_free_list_inc" (current value:
"32", data source: default, level: 9 dev/all, type: int)
Increment size of free lists (must be >= 1)
MCA btl: parameter "btl_openib_mpool" (current value: "grdma",
data source: default, level: 9 dev/all, type: string)
Name of the memory pool to be used (it is unlikely
that you will ever want to change this)
MCA btl: parameter "btl_openib_reg_mru_len" (current value:
"16", data source: default, level: 9 dev/all, type: int)
Length of the registration cache most recently used
list (must be >= 1)
MCA btl: parameter "btl_openib_cq_size" (current value:
"8192", data source: default, level: 9 dev/all, type: int, synonyms:
btl_openib_ib_cq_size)
Minimum size of the OpenFabrics completion queue (CQs
are automatically sized based on the number of peer MPI processes; this value
determines the *minimum* size of all CQs)
MCA btl: parameter "btl_openib_ib_cq_size" (current value:
"8192", data source: default, level: 9 dev/all, type: int, deprecated, synonym
of: btl_openib_cq_size)
Minimum size of the OpenFabrics completion queue (CQs
are automatically sized based on the number of peer MPI processes; this value
determines the *minimum* size of all CQs)
MCA btl: parameter "btl_openib_max_inline_data" (current
value: "-1", data source: default, level: 9 dev/all, type: int, synonyms:
btl_openib_ib_max_inline_data)
Maximum size of inline data segment (-1 = run-time
probe to discover max value, otherwise must be >= 0). If not explicitly set,
use max_inline_data from the INI file containing device-specific parameters
MCA btl: parameter "btl_openib_ib_max_inline_data" (current
value: "-1", data source: default, level: 9 dev/all, type: int, deprecated,
synonym of: btl_openib_max_inline_data)
Maximum size of inline data segment (-1 = run-time
probe to discover max value, otherwise must be >= 0). If not explicitly set,
use max_inline_data from the INI file containing device-specific parameters
MCA btl: parameter "btl_openib_pkey" (current value: "0", data
source: default, level: 9 dev/all, type: unsigned, synonyms:
btl_openib_ib_pkey_val)
OpenFabrics partition key (pkey) value. Unsigned
integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be
masked against the maximum allowable IB partition key value (0x7fff)
MCA btl: parameter "btl_openib_ib_pkey_val" (current value:
"0", data source: default, level: 9 dev/all, type: unsigned, deprecated,
synonym of: btl_openib_pkey)
OpenFabrics partition key (pkey) value. Unsigned
integer decimal or hex values are allowed (e.g., "3" or "0x3f") and will be
masked against the maximum allowable IB partition key value (0x7fff)
MCA btl: parameter "btl_openib_psn" (current value: "0", data
source: default, level: 9 dev/all, type: unsigned, synonyms: btl_openib_ib_psn)
OpenFabrics packet sequence starting number (must be
>= 0)
MCA btl: parameter "btl_openib_ib_psn" (current value: "0",
data source: default, level: 9 dev/all, type: unsigned, deprecated, synonym of:
btl_openib_psn)
OpenFabrics packet sequence starting number (must be
>= 0)
MCA btl: parameter "btl_openib_ib_qp_ous_rd_atom" (current
value: "4", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand outstanding atomic reads (must be >= 0)
MCA btl: parameter "btl_openib_mtu" (current value: "1k", data
source: default, level: 9 dev/all, type: int, synonyms: btl_openib_ib_mtu)
OpenFabrics MTU, in bytes (if not specified in INI
files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048
bytes, 5=4096 bytes
Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k",
5:"4k"
MCA btl: parameter "btl_openib_ib_mtu" (current value: "1k",
data source: default, level: 9 dev/all, type: int, deprecated, synonym of:
btl_openib_mtu)
OpenFabrics MTU, in bytes (if not specified in INI
files). Valid values are: 1=256 bytes, 2=512 bytes, 3=1024 bytes, 4=2048
bytes, 5=4096 bytes
Valid values: 1:"256B", 2:"512B", 3:"1k", 4:"2k",
5:"4k"
MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current
value: "25", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand minimum "receiver not ready" timer, in
seconds (must be >= 0 and <= 31)
MCA btl: parameter "btl_openib_ib_timeout" (current value:
"24", data source: environment, level: 9 dev/all, type: unsigned)
InfiniBand transmit timeout, plugged into formula:
4.096 microseconds * (2^btl_openib_ib_timeout) (must be >= 0 and <= 31)
MCA btl: parameter "btl_openib_ib_retry_count" (current value:
"7", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand transmit retry count (must be >= 0 and <=
7)
MCA btl: parameter "btl_openib_ib_rnr_retry" (current value:
"7", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand "receiver not ready" retry count; applies
*only* to SRQ/XRC queues. PP queues use RNR retry values of 0 because Open MPI
performs software flow control to guarantee that RNRs never occur (must be >= 0
and <= 7; 7 = "infinite")
MCA btl: parameter "btl_openib_ib_max_rdma_dst_ops" (current
value: "4", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand maximum pending RDMA destination
operations (must be >= 0)
MCA btl: parameter "btl_openib_ib_service_level" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
InfiniBand service level (must be >= 0 and <= 15)
MCA btl: parameter "btl_openib_ib_path_record_service_level"
(current value: "0", data source: default, level: 9 dev/all, type: unsigned)
Enable getting InfiniBand service level from
PathRecord (must be >= 0, 0 = disabled, positive = try to get the service level
from PathRecord)
MCA btl: parameter "btl_openib_use_eager_rdma" (current value:
"-1", data source: default, level: 9 dev/all, type: int)
Use RDMA for eager messages (-1 = use device default,
0 = do not use eager RDMA, 1 = use eager RDMA)
MCA btl: parameter "btl_openib_eager_rdma_threshold" (current
value: "16", data source: default, level: 9 dev/all, type: int)
Use RDMA for short messages after this number of
messages are received from a given peer (must be >= 1)
MCA btl: parameter "btl_openib_max_eager_rdma" (current value:
"16", data source: default, level: 9 dev/all, type: int)
Maximum number of peers allowed to use RDMA for short
messages (RDMA is used for all long messages, except if explicitly disabled,
such as with the "dr" pml) (must be >= 0)
MCA btl: parameter "btl_openib_eager_rdma_num" (current value:
"17", data source: default, level: 9 dev/all, type: int)
Number of RDMA buffers to allocate for small messages
(must be >= 1)
MCA btl: parameter "btl_openib_btls_per_lid" (current value:
"1", data source: default, level: 9 dev/all, type: unsigned)
Number of BTLs to create for each InfiniBand LID
(must be >= 1)
MCA btl: parameter "btl_openib_max_lmc" (current value: "1",
data source: default, level: 9 dev/all, type: unsigned)
Maximum number of LIDs to use for each device port
(must be >= 0, where 0 = use all available)
MCA btl: parameter "btl_openib_enable_apm_over_lmc" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Maximum number of alternative paths for each device
port (must be >= -1, where 0 = disable apm, -1 = all available alternative
paths )
MCA btl: parameter "btl_openib_enable_apm_over_ports" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Enable alternative path migration (APM) over
different ports of the same device (must be >= 0, where 0 = disable APM over
ports, 1 = enable APM over ports of the same device)
MCA btl: parameter "btl_openib_use_async_event_thread"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
If nonzero, use the thread that will handle
InfiniBand asynchronous events
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_enable_srq_resize" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Enable/Disable on demand SRQ resize. (0 = without
resizing, nonzero = with resizing)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_rroce_enable" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
Enable/Disable routing between different subnets(0 =
disable, nonzero = enable)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_buffer_alignment" (current
value: "64", data source: default, level: 9 dev/all, type: unsigned)
Preferred communication buffer alignment, in bytes
(must be > 0 and power of two)
MCA btl: parameter "btl_openib_cq_poll_ratio" (current value:
"100", data source: default, level: 9 dev/all, type: unsigned)
How often to poll high priority CQ versus low
priority CQ
MCA btl: parameter "btl_openib_eager_rdma_poll_ratio" (current
value: "100", data source: default, level: 9 dev/all, type: unsigned)
How often to poll eager RDMA channel versus CQ
MCA btl: parameter "btl_openib_hp_cq_poll_per_progress"
(current value: "10", data source: default, level: 9 dev/all, type: unsigned)
Max number of completion events to process for each
call of BTL progress engine
MCA btl: parameter "btl_openib_max_hw_msg_size" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
Maximum size (in bytes) of a single fragment of a
long message when using the RDMA protocols (must be > 0 and <= hw capabilities).
MCA btl: parameter "btl_openib_allow_max_memory_registration"
(current value: "true", data source: default, level: 9 dev/all, type: bool)
Allow maximum possible memory to register with HCA
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_memory_registration_verbose"
(current value: "0", data source: default, level: 9 dev/all, type: int)
Output some verbose memory registration information
(0 = no output, nonzero = output)
MCA btl: parameter "btl_openib_ignore_locality" (current
value: "0", data source: default, level: 9 dev/all, type: int)
Ignore any locality information and use all devices
(0 = use locality informaiton and use only close devices, nonzero = ignore
locality information)
MCA btl: informational "btl_openib_have_fork_support" (current
value: "true", data source: default, level: 9 dev/all, type: bool)
Whether the OpenFabrics stack supports applications
that invoke the "fork()" system call or not (0 = no, 1 = yes). Note that this
value does NOT indicate whether the system being run on supports "fork()" with
OpenFabrics applications or not.
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_exclusivity" (current value:
"1024", data source: default, level: 7 dev/basic, type: unsigned)
BTL exclusivity (must be >= 0)
MCA btl: parameter "btl_openib_flags" (current value: "310",
data source: default, level: 5 tuner/detail, type: unsigned)
BTL bit flags (general flags: SEND=1, PUT=2, GET=4,
SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only used by the
"dr" PML (ignored by others): ACK=16, CHECKSUM=32, RDMA_COMPLETION=128; flags
only used by the "bfo" PML (ignored by others): FAILOVER_SUPPORT=512)
MCA btl: parameter "btl_openib_rndv_eager_limit" (current
value: "12288", data source: default, level: 4 tuner/basic, type: size_t)
Size (in bytes, including header) of "phase 1"
fragment sent for all large messages (must be >= 0 and <= eager_limit)
MCA btl: parameter "btl_openib_eager_limit" (current value:
"12288", data source: default, level: 4 tuner/basic, type: size_t)
Maximum size (in bytes, including header) of "short"
messages (must be >= 1).
MCA btl: parameter "btl_openib_max_send_size" (current value:
"65536", data source: default, level: 4 tuner/basic, type: size_t)
Maximum size (in bytes) of a single "phase 2"
fragment of a long message when using the pipeline protocol (must be >= 1)
MCA btl: parameter "btl_openib_rdma_pipeline_send_length"
(current value: "1048576", data source: default, level: 4 tuner/basic, type:
size_t)
Length of the "phase 2" portion of a large message
(in bytes) when using the pipeline protocol. This part of the message will be
split into fragments of size max_send_size and sent using send/receive
semantics (must be >= 0; only relevant when the PUT flag is set)
MCA btl: parameter "btl_openib_rdma_pipeline_frag_size"
(current value: "1048576", data source: default, level: 4 tuner/basic, type:
size_t)
Maximum size (in bytes) of a single "phase 3"
fragment from a long message when using the pipeline protocol. These fragments
will be sent using RDMA semantics (must be >= 1; only relevant when the PUT
flag is set)
MCA btl: parameter "btl_openib_min_rdma_pipeline_size"
(current value: "1060864", data source: default, level: 4 tuner/basic, type:
size_t)
Messages smaller than this size (in bytes) will not
use the RDMA pipeline protocol. Instead, they will be split into fragments of
max_send_size and sent using send/receive semantics (must be >=0, and is
automatically adjusted up to at least
(eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT flag is
set)
MCA btl: parameter "btl_openib_latency" (current value: "4",
data source: default, level: 5 tuner/detail, type: unsigned)
Approximate latency of interconnect (0 = auto-detect
value at run-time [not supported in all BTL modules], >= 1 = latency in
microseconds)
MCA btl: parameter "btl_openib_bandwidth" (current value: "0",
data source: default, level: 5 tuner/detail, type: unsigned)
Approximate maximum bandwidth of interconnect (0 =
auto-detect value at run-time [not supported in all BTL modules], >= 1 =
bandwidth in Mbps)
MCA btl: parameter "btl_openib_receive_queues" (current value:
"P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64",
data source: default, level: 9 dev/all, type: string)
Colon-delimited, comma-delimited list of receive
queues: P,4096,8,6,4:P,32768,8,6,4
MCA btl: parameter "btl_openib_if_include" (current value: "",
data source: default, level: 9 dev/all, type: string)
Comma-delimited list of devices/ports to be used
(e.g. "mthca0,mthca1:2"; empty value means to use all ports found). Mutually
exclusive with btl_openib_if_exclude.
MCA btl: parameter "btl_openib_if_exclude" (current value: "",
data source: default, level: 9 dev/all, type: string)
Comma-delimited list of device/ports to be excluded
(empty value means to not exclude any ports). Mutually exclusive with
btl_openib_if_include.
MCA btl: parameter "btl_openib_ipaddr_include" (current value:
"", data source: default, level: 9 dev/all, type: string)
Comma-delimited list of IP Addresses to be used (e.g.
"192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_exclude.
MCA btl: parameter "btl_openib_ipaddr_exclude" (current value:
"", data source: default, level: 9 dev/all, type: string)
Comma-delimited list of IP Addresses to be excluded
(e.g. "192.168.1.0/24"). Mutually exclusive with btl_openib_ipaddr_include.
MCA btl: parameter "btl_openib_gid_index" (current value: "0",
data source: default, level: 9 dev/all, type: int)
GID index to use on verbs device ports
MCA btl: parameter "btl_openib_memalign" (current value: "32",
data source: default, level: 9 dev/all, type: int)
[64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0)
memoryalignment for all malloc calls if btl openib is used.
MCA btl: parameter "btl_openib_memalign_threshold" (current
value: "12288", data source: default, level: 9 dev/all, type: size_t)
Allocating memory more than
btl_openib_memalign_threshholdbytes will automatically be aligned to the value
of btl_openib_memalign bytes.memalign_threshhold defaults to the same value as
mca_btl_openib_eager_limit.
MCA btl: parameter "btl_openib_cpc_include" (current value:
"", data source: default, level: 9 dev/all, type: string)
Method used to select OpenFabrics connections (valid
values: rdmacm,udcm)
MCA btl: parameter "btl_openib_cpc_exclude" (current value:
"", data source: default, level: 9 dev/all, type: string)
Method used to exclude OpenFabrics connections (valid
values: rdmacm,udcm)
MCA btl: parameter "btl_openib_connect_rdmacm_priority"
(current value: "30", data source: default, level: 9 dev/all, type: int)
The selection method priority for rdma_cm
MCA btl: parameter "btl_openib_connect_rdmacm_port" (current
value: "0", data source: default, level: 9 dev/all, type: unsigned)
The selection method port for rdma_cm
MCA btl: parameter "btl_openib_connect_rdmacm_resolve_timeout"
(current value: "30000", data source: default, level: 9 dev/all, type: int)
The timeout (in miliseconds) for address and route
resolution
MCA btl: parameter "btl_openib_connect_rdmacm_retry_count"
(current value: "20", data source: default, level: 9 dev/all, type: int)
Maximum number of times rdmacm will retry route
resolution
MCA btl: parameter
"btl_openib_connect_rdmacm_reject_causes_connect_error" (current value:
"false", data source: default, level: 9 dev/all, type: bool)
The drivers for some devices are buggy such that an
RDMA REJECT action may result in a CONNECT_ERROR event instead of a REJECTED
event. Setting this MCA parameter to true tells Open MPI to treat
CONNECT_ERROR events on connections where a REJECT is expected as a REJECT
(default: false)
Valid values: 0: f|false|disabled, 1: t|true|enabled
MCA btl: parameter "btl_openib_connect_udcm_priority" (current
value: "63", data source: default, level: 9 dev/all, type: int)
Priority of the udcm connection method
MCA btl: parameter "btl_openib_connect_udcm_recv_count"
(current value: "512", data source: default, level: 9 dev/all, type: int)
Number of registered buffers to post
MCA btl: parameter "btl_openib_connect_udcm_timeout" (current
value: "500000", data source: default, level: 9 dev/all, type: int)
Ack timeout for udcm connection messages
MCA btl: parameter "btl_openib_connect_udcm_max_retry"
(current value: "25", data source: default, level: 9 dev/all, type: int)
Maximum number of times to retry sending a udcm
connection message
smime.p7s
Description: S/MIME Cryptographic Signature
