[lustre-discuss] need to always manually add network after reboot
G'Day all, I'm finding that when I reboot any node in our new HPC, I need to keep manually adding the network using lnetctl net add --net tcp --if ens2f0 Then I can do an lnetctl net show and see the tcp part active... I have options in /etc/modprobe.d/lnet.conf options lnet networks=tcp and [root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf options lnet networks="tcp(ens2f0)" options lnet ip2nets="tcp(ens2f0) 10.140.93.* I've read the doco and tried to understand the correct parameters for a simple Lustre config so this is what I worked out is needed... but I suspect its still wrong. Any help appreciated :) Sid Young ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] need to always manually add network after reboot
Hello If I understand correctly, you're telling that you have 2 configuration files: /etc/modprobe.d/lnet.conf options lnet networks=tcp [root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf options lnet networks="tcp(ens2f0)" options lnet ip2nets="tcp(ens2f0) 10.140.93.* That means you are declaring twice the "networks" option for "lnet" kernel module. I don't know how 'modprobe' will behave regarding that. If you have a very simple configuration, where your nodes only have one Ethernet interface "ens2f0", you only need the following lines, from the 3 above: options lnet networks="tcp(ens2f0)" If this interface is the only Ethernet interface on your host, you don't even need a network specific setup. By default, when loading Lustre, in the absence of a network configuration, Lustre will automatically setup the only ethernet interface to use it for "tcp". Aurélien De : lustre-discuss au nom de Sid Young via lustre-discuss Répondre à : Sid Young Date : mardi 23 février 2021 à 06:59 À : lustre-discuss Objet : [EXTERNAL] [lustre-discuss] need to always manually add network after reboot CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. G'Day all, I'm finding that when I reboot any node in our new HPC, I need to keep manually adding the network using lnetctl net add --net tcp --if ens2f0 Then I can do an lnetctl net show and see the tcp part active... I have options in /etc/modprobe.d/lnet.conf options lnet networks=tcp and [root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf options lnet networks="tcp(ens2f0)" options lnet ip2nets="tcp(ens2f0) 10.140.93.* I've read the doco and tried to understand the correct parameters for a simple Lustre config so this is what I worked out is needed... but I suspect its still wrong. Any help appreciated :) Sid Young ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] need to always manually add network after reboot
Hi Sid, Notice that you are using lnetctl net add to add the lnet network, which means you should be using a recent version of Lustre that depends on /etc/lnet.conf for boot time lnet configuration. You can save the current lnet configuration using command: lnetctl export --backup > /etc/lnet.conf (make a backup of the original file first if required) On next boot, lnet.service will load your lnet configuration from the file. Or you can manually build lnet.conf as lnetctl seems to have occasion problems with some of the fields exported by "lnetctl export --backup" Attaching my simple lnet.conf for your reference: # cat /etc/lnet.conf ip2nets: - net-spec: o2ib ip-range: 0: 10.2.8.* - net-spec: tcp ip-range: 0: 10.5.9.* route: - net: o2ib gateway: 10.5.9.25@tcp hop: -1 priority: 0 - net: o2ib gateway: 10.5.9.24@tcp hop: -1 priority: 0 global: numa_range: 0 max_intf: 200 discovery: 1 drop_asym_route: 0 Best regards, Angelos On 23/02/2021 13:58, Sid Young via lustre-discuss wrote: G'Day all, I'm finding that when I reboot any node in our new HPC, I need to keep manually adding the network using lnetctl net add --net tcp --if ens2f0 Then I can do an lnetctl net show and see the tcp part active... I have options in /etc/modprobe.d/lnet.conf options lnet networks=tcp and [root@hpc-oss-03 ~]# cat /etc/modprobe.d/lustre.conf options lnet networks="tcp(ens2f0)" options lnet ip2nets="tcp(ens2f0) 10.140.93.* I've read the doco and tried to understand the correct parameters for a simple Lustre config so this is what I worked out is needed... but I suspect its still wrong. Any help appreciated :) Sid Young ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- Angelos Ching ClusterTech Limited Tel : +852-2655-6138 Fax : +852-2994-2101 Address : Unit 211-213, Lakeside 1, 8 Science Park West Ave., Shatin, Hong Kong Got praises or room for improvements? http://bit.ly/TellAngelos The information contained in this e-mail and its attachments is confidential and intended solely for the specified addressees. If you have received this email in error, please do not read, copy, distribute, disclose or use any information of this email in any way and please immediately notify the sender and delete this email. Thank you for your cooperation. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
