Thanks Rick. I removed my lnet modprobe options and adapted my lnet.conf file
to:
# cat /etc/lnet.conf
ip2nets:
- net-spec: o2ib1
interfaces:
0: ib0
global:
discovery: 0
#
Now "lnetctl export" doesn't have any reference to NIDs on the other networks,
so that's good. However, I'm still seeing some values that concern me:
# lnetctl export | grep -e Multi -e discover | sort -u
discovery: 1
Multi-Rail: True
#
Any idea why discovery is still 1 if I'm specifying that to 0 in the lnet.conf
file? I'm a little concerned that with Multi-Rail still True and discovery on,
the client could still find its way back to the TCP route.
From: Riccardo Veraldi <riccardo.vera...@cnaf.infn.it>
Date: Monday, September 13, 2021 at 3:16 PM
To: "Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]"
<darby.vicke...@nasa.gov>, "lustre-discuss@lists.lustre.org"
<lustre-discuss@lists.lustre.org>
Subject: [EXTERNAL] Re: [lustre-discuss] Disabling multi-rail dynamic discovery
I would use configuration on /etc/lnet.conf and I would not use anymore the
older style configuration in
/etc/modprobe.d/lustre.conf
for example in my /etc/lnet.conf configuration I have:
ip2nets:
- net-spec: o2ib
interfaces:
0: ib0
- net-spec: tcp
interfaces:
0: enp24s0f0
global:
discovery: 0
As I disabled the auto discovery.
Regarding ko2ib you can just use /etc/modprobe.d/ko2iblnd.conf
Mine looks like this:
options ko2iblnd peer_credits=128 peer_credits_hiw=64 credits=1024 ntx=2048
map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1
conns_per_peer=4
Hope it helps.
Rick
On 9/13/21 1:53 PM, Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.] via
lustre-discuss wrote:
Hello,
I would like to know how to turn off auto discovery of peers on a client. This
seems like it should be straight forward but we can't get it to work. Please
fill me in on what I'm missing.
We recently upgraded our servers to 2.14. Our servers are multi-homed (1 tcp
network and 2 separate IB networks) but we want them to be single rail. On one
of our clusters we are still using the 2.12.6 client and it uses one of the IB
networks for lustre. The modprobe file from one of the client nodes:
# cat /etc/modprobe.d/lustre.conf
options lnet networks=o2ib1(ib0)
options ko2iblnd map_on_demand=32
#
The client does have a route to the TCP network. This is intended to allow
jobs on the compute nodes to access licenese servers, not for any serious I/O.
We recently discovered that due to some instability in the IB fabric, the
client was trying to fail over to tcp:
# dmesg | grep Lustre
[ 250.205912] Lustre: Lustre: Build Version: 2.12.6
[ 255.886086] Lustre: Mounted scratch-client
[ 287.247547] Lustre: 3472:0:(client.c:2146:ptlrpc_expire_one_request()) @@@
Request sent has timed out for sent delay: [sent 1630699139/real 0]
req@ffff98deb9358480 x1709911947878336/t0(0)
o9->hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4<mailto:hpfs-fsl-OST0001-osc-ffff9880cfb80000@192.52.98.33@tcp:28/4>
lens 224/224 e 0 to 1 dl 1630699145 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[ 739.832744] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request()) @@@
Request sent has timed out for sent delay: [sent 1630699591/real 0]
req@ffff98deb935da00 x1709911947883520/t0(0)
o400->scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10<mailto:scratch-MDT0000-mdc-ffff98b0f1fc0800@192.52.98.31@tcp:12/10>
lens 224/224 e 0 to 1 dl 1630699598 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[ 739.832755] Lustre: 3526:0:(client.c:2146:ptlrpc_expire_one_request())
Skipped 5 previous similar messages
[ 739.832762] LustreError: 166-1: MGC10.150.100.30@o2ib1: Connection to MGS
(at 192.52.98.30@tcp) was lost; in progress operations using this service will
fail
[ 739.832769] Lustre: hpfs-fsl-MDT0000-mdc-ffff9880cfb80000: Connection to
hpfs-fsl-MDT0000 (at 192.52.98.30@tcp) was lost; in progress operations using
this service will wait for recovery to complete
[ 1090.978619] LustreError: 167-0: scratch-MDT0000-mdc-ffff98b0f1fc0800: This
client was evicted by scratch-MDT0000; in progress operations using this
service will fail.
I'm pretty sure this is due to the auto discovery. Again, from a client:
# lnetctl export | grep -e Multi -e discover | sort -u
discovery: 0
Multi-Rail: True
#
We want to restrict lustre to only the IB NID but its not clear exactly how to
do that.
Here is one attempt:
[root@r1i1n18 lnet]# service lustre3 stop
Shutting down lustre mounts
Lustre modules successfully unloaded
[root@r1i1n18 lnet]# lsmod | grep lnet
[root@r1i1n18 lnet]# cat /etc/lnet.conf
global:
discovery: 0
[root@r1i1n18 lnet]# service lustre3 start
Mounting /ephemeral... done.
Mounting /nobackup... done.
[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u
discovery: 1
Multi-Rail: True
[root@r1i1n18 lnet]#
And a similar attempt (same lnet.conf file), but trying to turn off the
discovery before doing the mounts:
[root@r1i1n18 lnet]# service lustre3 stop
Shutting down lustre mounts
Lustre modules successfully unloaded
[root@r1i1n18 lnet]# modprobe lnet
[root@r1i1n18 lnet]# lnetctl set discovery 0
[root@r1i1n18 lnet]# service lustre3 start
Mounting /ephemeral... done.
Mounting /nobackup... done.
[root@r1i1n18 lnet]# lnetctl export | grep -e Multi -e discover | sort -u
discovery: 0
Multi-Rail: True
[root@r1i1n18 lnet]#
If someone can point me in the right direction, I'd appreciate it.
Thanks,
Darby
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=04%7C01%7Cdarby.vicker-1%40nasa.gov%7Cb2a81e07db45418e29df08d976fbbebd%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637671645889714604%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oRqJx%2FcdY29ppvvulneydVZZY%2Frm1vD8EddtDofafgk%3D&reserved=0>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
