Hi Eli, Yes & no; part of my info is a bit rusty because I carried them from version around 2.10. MR is now turned on by default.
But you'll need to have an IP setup on each IPoIB interface, and for all ib0 & all ib1 interface, they should be in different subnet. Eg: all ib0 on 192.168.100.0/24 and all ib1 on 192.168.101.0/24 Then you'll setup 2 NID for each node; around 2.10 when WhamCloud started implementing MR, they have added lnetctl and documentation updated to recommend against setting up lnet via modprobe. When setting up via lnetctl or lnet.conf, you don't need to specify an IP for each NID, but you still have to setup an NID for each of your physical interface. You may want to look at the ip2net option as well. Best regards, Angelos (Sent from mobile, please pardon me for typos and cursoriness.) > 17/1/2022 0:39、E.S. Rosenberg <esr+lus...@mail.hebrew.edu>のメール: > > > Hi Angelos, > Thanks for the answer! > According to the lecture Lauz just sent multirail is on by default no? > So I don't need to set IPs on each interface (using > interfaces/netplan/network scripts) just a single NID at the lnet level? > Thanks again, > Eli > >> On Sun, Jan 16, 2022 at 5:20 PM Angelos Ching <angelosch...@clustertech.com> >> wrote: >> Hi Eli, >> >> I think you can leave out the IP so long as you specific the interface, but >> IIRC you can't setup multirail via modprobe, you should setup Lnet using >> lnetctl >> >> Best regards, >> Angelos >> (Sent from mobile, please pardon me for typos and cursoriness.) >> >>> 16/1/2022 21:38、E.S. Rosenberg <esr+lus...@mail.hebrew.edu>のメール: >>> >>> >>> Hey Lauz, >>> >>> Thanks for the reply! >>> Indeed it sounds like multirail is what I am referring to (my wrong memory >>> of multirail was a failover technique allowing connection of both TCP and >>> IPoIB or multiple IPoIB between nodes at the same time), but I am still >>> left with a question. >>> In the modprobe arguments for lnet I'm binding all the interfaces into a >>> single network but I don't define IPs for the interfaces that is done at >>> the OS level. >>> Lustre as far as I recall does use IPoIB so IPs are needed or could I even >>> leave out the IP definitions and let lustre figure out what it wants there? >>> It just seems kind of excessive and adding points of failure that a node >>> with 4 dual port IB cards would end up needing 8 IP addresses... >>> Thanks again, >>> Eli >>> >>> >>> >>>> On Sun, Jan 16, 2022 at 2:32 PM Laurence Horrocks-Barlow >>>> <laure...@qsplace.co.uk> wrote: >>>> The limit of IPoIB is active/backup when using traditional bonding, >>>> however I believe you are wanting to multirail your IB. This is achieved >>>> buy using multiple LNet's (assuming it uses the same fabric), you should >>>> be able to configure for active/active. >>>> >>>> https://wiki.whamcloud.com/display/LNet/Multi-Rail+Overview >>>> >>>> This should help with most of the concepts. >>>> >>>> -- Lauz >>>> >>>>> On 16 January 2022 11:56:16 GMT, "E.S. Rosenberg" >>>>> <esr+lus...@mail.hebrew.edu> wrote: >>>>> Hey everyone, >>>>> >>>>> This is probably off-topic but I can't find any documents on the subject >>>>> and since Lustre uses IPoIB I suspect others here have dealt with this >>>>> question. >>>>> >>>>> If I have a node connected with multiple IB links should each connected >>>>> IB port have it's own IP address or is there a way similar to LACP on the >>>>> Ethernet side to bond all the links and use only a single IP address to >>>>> refer to the node? And what is the better method? >>>>> >>>>> In the past I never had this luxury, but now I'm starting a small new >>>>> cluster currently made up of a few GPU nodes and a Lustre filesystem so >>>>> plenty of IB ports to go around. >>>>> >>>>> Thanks! >>>>> Eli >>> _______________________________________________ >>> lustre-discuss mailing list >>> lustre-discuss@lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
