Thanks. That /is/ one solution, and what I’ll do in the interim since this has
to work in at least some fashion, but I would actually like to use UCX if
OpenIB is going to be deprecated. How do I find out what’s actually wrong?
--
#BlackLivesMatter
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novos...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'
> On Jul 29, 2021, at 11:35 AM, Ralph Castain via users
> <users@lists.open-mpi.org> wrote:
>
> So it _is_ UCX that is the problem! Try using OMPI_MCA_pml=ob1 instead
>
>> On Jul 29, 2021, at 8:33 AM, Ryan Novosielski <novos...@rutgers.edu> wrote:
>>
>> Thanks, Ralph. This /does/ change things, but not very much. I was not under
>> the impression that I needed to do that, since when I ran without having
>> built against UCX, it warned me about the openib method being deprecated. By
>> default, does OpenMPI not use either anymore, and I need to specifically
>> call for UCX? Seems strange.
>>
>> Anyhow, I’ve got some variables defined still, in addition to your
>> suggestion, for verbosity:
>>
>> [novosirj@amarel-test2 ~]$ env | grep ^OMPI
>> OMPI_MCA_pml=ucx
>> OMPI_MCA_opal_common_ucx_opal_mem_hooks=1
>> OMPI_MCA_pml_ucx_verbose=100
>>
>> Here goes:
>>
>> [novosirj@amarel-test2 ~]$ srun -n 2 --mpi=pmi2 -p oarc --reservation=UCX
>> ./mpihello-gcc-8-openmpi-4.0.6
>> srun: job 13995650 queued and waiting for resources
>> srun: job 13995650 has been allocated resources
>> --------------------------------------------------------------------------
>> WARNING: There was an error initializing an OpenFabrics device.
>>
>> Local host: gpu004
>> Local device: mlx4_0
>> --------------------------------------------------------------------------
>> --------------------------------------------------------------------------
>> WARNING: There was an error initializing an OpenFabrics device.
>>
>> Local host: gpu004
>> Local device: mlx4_0
>> --------------------------------------------------------------------------
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using OPAL
>> memory hooks as external events
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using OPAL
>> memory hooks as external events
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:197
>> mca_pml_ucx_open: UCX version 1.5.2
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:197
>> mca_pml_ucx_open: UCX version 1.5.2
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 self/self:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/eno1:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/ib0:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 self/self:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>> rc/mlx4_0:1: did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>> ud/mlx4_0:1: did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/sysv:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/posix:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 cma/cma:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29823]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:311 support
>> level is none
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/eno1:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/ib0:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>> rc/mlx4_0:1: did not match transport list
>> --------------------------------------------------------------------------
>> No components were able to be opened in the pml framework.
>>
>> This typically means that either no components of this type were
>> installed, or none of the installed components can be loaded.
>> Sometimes this means that shared libraries required by these
>> components are unable to be found/loaded.
>>
>> Host: gpu004
>> Framework: pml
>> --------------------------------------------------------------------------
>> [gpu004.amarel.rutgers.edu:29823] PML ucx cannot be selected
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>> ud/mlx4_0:1: did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/sysv:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/posix:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 cma/cma:
>> did not match transport list
>> [gpu004.amarel.rutgers.edu:29824]
>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:311 support
>> level is none
>> --------------------------------------------------------------------------
>> No components were able to be opened in the pml framework.
>>
>> This typically means that either no components of this type were
>> installed, or none of the installed components can be loaded.
>> Sometimes this means that shared libraries required by these
>> components are unable to be found/loaded.
>>
>> Host: gpu004
>> Framework: pml
>> --------------------------------------------------------------------------
>> [gpu004.amarel.rutgers.edu:29824] PML ucx cannot be selected
>> slurmstepd: error: *** STEP 13995650.0 ON gpu004 CANCELLED AT
>> 2021-07-29T11:31:19 ***
>> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
>> srun: error: gpu004: tasks 0-1: Exited with exit code 1
>>
>> --
>> #BlackLivesMatter
>> ____
>> || \\UTGERS,
>> |---------------------------*O*---------------------------
>> ||_// the State | Ryan Novosielski - novos...@rutgers.edu
>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>> || \\ of NJ | Office of Advanced Research Computing - MSB C630,
>> Newark
>> `'
>>
>>> On Jul 29, 2021, at 8:34 AM, Ralph Castain via users
>>> <users@lists.open-mpi.org> wrote:
>>>
>>> Ryan - I suspect what Sergey was trying to say was that you need to ensure
>>> OMPI doesn't try to use the OpenIB driver, or at least that it doesn't
>>> attempt to initialize it. Try adding
>>>
>>> OMPI_MCA_pml=ucx
>>>
>>> to your environment.
>>>
>>>
>>>> On Jul 29, 2021, at 1:56 AM, Sergey Oblomov via users
>>>> <users@lists.open-mpi.org> wrote:
>>>>
>>>> Hi
>>>>
>>>> This issue arrives from BTL OpenIB, not related to UCX
>>>>
>>>> From: users <users-boun...@lists.open-mpi.org> on behalf of Ryan
>>>> Novosielski via users <users@lists.open-mpi.org>
>>>> Date: Thursday, 29 July 2021, 08:25
>>>> To: users@lists.open-mpi.org <users@lists.open-mpi.org>
>>>> Cc: Ryan Novosielski <novos...@rutgers.edu>
>>>> Subject: [OMPI users] OpenMPI 4.0.6 w/GCC 8.5 on CentOS 7.9; "WARNING:
>>>> There was an error initializing an OpenFabrics device."
>>>>
>>>> Hi there,
>>>>
>>>> New to using UCX, as a result of having built OpenMPI without it and
>>>> running tests and getting warned. Installed UCX from the distribution:
>>>>
>>>> [novosirj@amarel-test2 ~]$ rpm -qa ucx
>>>> ucx-1.5.2-1.el7.x86_64
>>>>
>>>> …and rebuilt OpenMPI. Built fine. However, I’m getting some pretty
>>>> unhelpful messages about not using the IB card. I looked around the
>>>> internet some and set a couple of environment variables to get a little
>>>> more information:
>>>>
>>>> OMPI_MCA_opal_common_ucx_opal_mem_hooks=1
>>>> export OMPI_MCA_pml_ucx_verbose=100
>>>>
>>>> Here’s what happens:
>>>>
>>>> [novosirj@amarel-test2 ~]$ srun -n 2 --mpi=pmi2 -p oarc --reservation=UCX
>>>> ./mpihello-gcc-8-openmpi-4.0.6
>>>> srun: job 13993927 queued and waiting for resources
>>>> srun: job 13993927 has been allocated resources
>>>> --------------------------------------------------------------------------
>>>> WARNING: There was an error initializing an OpenFabrics device.
>>>>
>>>> Local host: gpu004
>>>> Local device: mlx4_0
>>>> --------------------------------------------------------------------------
>>>> --------------------------------------------------------------------------
>>>> WARNING: There was an error initializing an OpenFabrics device.
>>>>
>>>> Local host: gpu004
>>>> Local device: mlx4_0
>>>> --------------------------------------------------------------------------
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using
>>>> OPAL memory hooks as external events
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:197
>>>> mca_pml_ucx_open: UCX version 1.5.2
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using
>>>> OPAL memory hooks as external events
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:197
>>>> mca_pml_ucx_open: UCX version 1.5.2
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> self/self: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> tcp/eno1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> self/self: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/ib0:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> rc/mlx4_0:1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> ud/mlx4_0:1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/sysv:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> mm/posix: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 cma/cma:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:311 support
>>>> level is none
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:268
>>>> mca_pml_ucx_close
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> tcp/eno1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 tcp/ib0:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> rc/mlx4_0:1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> ud/mlx4_0:1: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 mm/sysv:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304
>>>> mm/posix: did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:304 cma/cma:
>>>> did not match transport list
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:311 support
>>>> level is none
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/ompi/mca/pml/ucx/pml_ucx.c:268
>>>> mca_pml_ucx_close
>>>> [gpu004.amarel.rutgers.edu:02326]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using
>>>> OPAL memory hooks as external events
>>>> [gpu004.amarel.rutgers.edu:02327]
>>>> ../../../../../openmpi-4.0.6/opal/mca/common/ucx/common_ucx.c:147 using
>>>> OPAL memory hooks as external events
>>>> Hello world from processor gpu004.amarel.rutgers.edu, rank 0 out of 2
>>>> processors
>>>> Hello world from processor gpu004.amarel.rutgers.edu, rank 1 out of 2
>>>> processors
>>>>
>>>> Here’s the output of a couple more commands that seem to be recommended
>>>> when looking into this:
>>>>
>>>> [novosirj@gpu004 ~]$ ucx_info -d
>>>> #
>>>> # Memory domain: self
>>>> # component: self
>>>> # register: unlimited, cost: 0 nsec
>>>> # remote key: 8 bytes
>>>> #
>>>> # Transport: self
>>>> #
>>>> # Device: self
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6911.00 MB/sec
>>>> # latency: 0 nsec
>>>> # overhead: 10 nsec
>>>> # put_short: <= 4294967295
>>>> # put_bcopy: unlimited
>>>> # get_bcopy: unlimited
>>>> # am_short: <= 8k
>>>> # am_bcopy: <= 8k
>>>> # domain: cpu
>>>> # atomic_add: 32, 64 bit
>>>> # atomic_and: 32, 64 bit
>>>> # atomic_or: 32, 64 bit
>>>> # atomic_xor: 32, 64 bit
>>>> # atomic_fadd: 32, 64 bit
>>>> # atomic_fand: 32, 64 bit
>>>> # atomic_for: 32, 64 bit
>>>> # atomic_fxor: 32, 64 bit
>>>> # atomic_swap: 32, 64 bit
>>>> # atomic_cswap: 32, 64 bit
>>>> # connection: to iface
>>>> # priority: 0
>>>> # device address: 0 bytes
>>>> # iface address: 8 bytes
>>>> # error handling: none
>>>> #
>>>> #
>>>> # Memory domain: tcp
>>>> # component: tcp
>>>> #
>>>> # Transport: tcp
>>>> #
>>>> # Device: eno1
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 113.16 MB/sec
>>>> # latency: 5776 nsec
>>>> # overhead: 50000 nsec
>>>> # am_bcopy: <= 8k
>>>> # connection: to iface
>>>> # priority: 1
>>>> # device address: 4 bytes
>>>> # iface address: 2 bytes
>>>> # error handling: none
>>>> #
>>>> # Device: ib0
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6239.81 MB/sec
>>>> # latency: 5210 nsec
>>>> # overhead: 50000 nsec
>>>> # am_bcopy: <= 8k
>>>> # connection: to iface
>>>> # priority: 1
>>>> # device address: 4 bytes
>>>> # iface address: 2 bytes
>>>> # error handling: none
>>>> #
>>>> #
>>>> # Memory domain: ib/mlx4_0
>>>> # component: ib
>>>> # register: unlimited, cost: 90 nsec
>>>> # remote key: 16 bytes
>>>> # local memory handle is required for zcopy
>>>> #
>>>> # Transport: rc
>>>> #
>>>> # Device: mlx4_0:1
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6433.22 MB/sec
>>>> # latency: 900 nsec + 1 * N
>>>> # overhead: 75 nsec
>>>> # put_short: <= 88
>>>> # put_bcopy: <= 8k
>>>> # put_zcopy: <= 1g, up to 6 iov
>>>> # put_opt_zcopy_align: <= 512
>>>> # put_align_mtu: <= 2k
>>>> # get_bcopy: <= 8k
>>>> # get_zcopy: 33..1g, up to 6 iov
>>>> # get_opt_zcopy_align: <= 512
>>>> # get_align_mtu: <= 2k
>>>> # am_short: <= 87
>>>> # am_bcopy: <= 8191
>>>> # am_zcopy: <= 8191, up to 5 iov
>>>> # am_opt_zcopy_align: <= 512
>>>> # am_align_mtu: <= 2k
>>>> # am header: <= 127
>>>> # domain: device
>>>> # connection: to ep
>>>> # priority: 10
>>>> # device address: 3 bytes
>>>> # ep address: 4 bytes
>>>> # error handling: peer failure
>>>> #
>>>> #
>>>> # Transport: ud
>>>> #
>>>> # Device: mlx4_0:1
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6433.22 MB/sec
>>>> # latency: 910 nsec
>>>> # overhead: 105 nsec
>>>> # am_short: <= 172
>>>> # am_bcopy: <= 4088
>>>> # am_zcopy: <= 4088, up to 7 iov
>>>> # am_opt_zcopy_align: <= 512
>>>> # am_align_mtu: <= 4k
>>>> # am header: <= 3984
>>>> # connection: to ep, to iface
>>>> # priority: 10
>>>> # device address: 3 bytes
>>>> # iface address: 3 bytes
>>>> # ep address: 6 bytes
>>>> # error handling: peer failure
>>>> #
>>>> #
>>>> # Memory domain: rdmacm
>>>> # component: rdmacm
>>>> # supports client-server connection establishment via sockaddr
>>>> # < no supported devices found >
>>>> #
>>>> # Memory domain: sysv
>>>> # component: sysv
>>>> # allocate: unlimited
>>>> # remote key: 32 bytes
>>>> #
>>>> # Transport: mm
>>>> #
>>>> # Device: sysv
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6911.00 MB/sec
>>>> # latency: 80 nsec
>>>> # overhead: 10 nsec
>>>> # put_short: <= 4294967295
>>>> # put_bcopy: unlimited
>>>> # get_bcopy: unlimited
>>>> # am_short: <= 92
>>>> # am_bcopy: <= 8k
>>>> # domain: cpu
>>>> # atomic_add: 32, 64 bit
>>>> # atomic_and: 32, 64 bit
>>>> # atomic_or: 32, 64 bit
>>>> # atomic_xor: 32, 64 bit
>>>> # atomic_fadd: 32, 64 bit
>>>> # atomic_fand: 32, 64 bit
>>>> # atomic_for: 32, 64 bit
>>>> # atomic_fxor: 32, 64 bit
>>>> # atomic_swap: 32, 64 bit
>>>> # atomic_cswap: 32, 64 bit
>>>> # connection: to iface
>>>> # priority: 0
>>>> # device address: 8 bytes
>>>> # iface address: 16 bytes
>>>> # error handling: none
>>>> #
>>>> #
>>>> # Memory domain: posix
>>>> # component: posix
>>>> # allocate: unlimited
>>>> # remote key: 37 bytes
>>>> #
>>>> # Transport: mm
>>>> #
>>>> # Device: posix
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 6911.00 MB/sec
>>>> # latency: 80 nsec
>>>> # overhead: 10 nsec
>>>> # put_short: <= 4294967295
>>>> # put_bcopy: unlimited
>>>> # get_bcopy: unlimited
>>>> # am_short: <= 92
>>>> # am_bcopy: <= 8k
>>>> # domain: cpu
>>>> # atomic_add: 32, 64 bit
>>>> # atomic_and: 32, 64 bit
>>>> # atomic_or: 32, 64 bit
>>>> # atomic_xor: 32, 64 bit
>>>> # atomic_fadd: 32, 64 bit
>>>> # atomic_fand: 32, 64 bit
>>>> # atomic_for: 32, 64 bit
>>>> # atomic_fxor: 32, 64 bit
>>>> # atomic_swap: 32, 64 bit
>>>> # atomic_cswap: 32, 64 bit
>>>> # connection: to iface
>>>> # priority: 0
>>>> # device address: 8 bytes
>>>> # iface address: 16 bytes
>>>> # error handling: none
>>>> #
>>>> #
>>>> # Memory domain: cma
>>>> # component: cma
>>>> # register: unlimited, cost: 9 nsec
>>>> #
>>>> # Transport: cma
>>>> #
>>>> # Device: cma
>>>> #
>>>> # capabilities:
>>>> # bandwidth: 11145.00 MB/sec
>>>> # latency: 80 nsec
>>>> # overhead: 400 nsec
>>>> # put_zcopy: unlimited, up to 16 iov
>>>> # put_opt_zcopy_align: <= 1
>>>> # put_align_mtu: <= 1
>>>> # get_zcopy: unlimited, up to 16 iov
>>>> # get_opt_zcopy_align: <= 1
>>>> # get_align_mtu: <= 1
>>>> # connection: to iface
>>>> # priority: 0
>>>> # device address: 8 bytes
>>>> # iface address: 4 bytes
>>>> # error handling: none
>>>> #
>>>>
>>>> [novosirj@gpu004 ~]$ ucx_info -p -u t
>>>> #
>>>> # UCP context
>>>> #
>>>> # md 0 : self
>>>> # md 1 : tcp
>>>> # md 2 : ib/mlx4_0
>>>> # md 3 : rdmacm
>>>> # md 4 : sysv
>>>> # md 5 : posix
>>>> # md 6 : cma
>>>> #
>>>> # resource 0 : md 0 dev 0 flags -- self/self
>>>> # resource 1 : md 1 dev 1 flags -- tcp/eno1
>>>> # resource 2 : md 1 dev 2 flags -- tcp/ib0
>>>> # resource 3 : md 2 dev 3 flags -- rc/mlx4_0:1
>>>> # resource 4 : md 2 dev 3 flags -- ud/mlx4_0:1
>>>> # resource 5 : md 3 dev 4 flags -s rdmacm/sockaddr
>>>> # resource 6 : md 4 dev 5 flags -- mm/sysv
>>>> # resource 7 : md 5 dev 6 flags -- mm/posix
>>>> # resource 8 : md 6 dev 7 flags -- cma/cma
>>>> #
>>>> # memory: 0.84MB, file descriptors: 2
>>>> # create time: 5.032 ms
>>>> #
>>>>
>>>> Thanks for any help you can offer. What am I missing?
>>>>
>>>> --
>>>> #BlackLivesMatter
>>>> ____
>>>> || \\UTGERS,
>>>> |---------------------------*O*---------------------------
>>>> ||_// the State | Ryan Novosielski - novos...@rutgers.edu
>>>> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
>>>> || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
>>>> `'
>>>>
>>>
>>
>
>
