I have a very similar setup, and actually I would suggest to use the 1Gb interface for PXE booting, as well as for all the other administrative tasks, to keep as much unnecessary traffic off your 10 Gb network. Which interface you use for PXE booting has very little to do with which NICs are getting configured later, or how traffic is routed later.
I'm using the ip and nicips.eth0 and nicips.eth2 attributes on the node to configure the correct IP addresses that I want. Also, be sure to use nicextraparams.eth2=MTU=9000 if you want Jumbo frames on your 10 GB network. Finally, I'm using the nichostnamesuffixes attribute so the nodes have different, but predictable, names on the 1 Gb network and the 10 Gb network. The other important thing is to set the gateway on only one network - probably the 1 Gb network, so the 10 Gb network remains strictly for internal traffic. Of course, you will also need to tell higher-level software (in our case, OpenMPI and Torque) when to use the 1 Gb and when to use the 10 Gb network. That's easily done thanks to the distinguishable DNS names. _______________________________________________________________________ Kevin Keane | Systems Architect | University of San Diego ITS | [email protected] Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859 *REMEMBER! **No one from IT at USD will ever ask to confirm or supply your password*. These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them! On Tue, Feb 12, 2019 at 6:20 AM Thomas HUMMEL <[email protected]> wrote: > Hello, > > I'm using xCAT 2.11 in production (I know it's an old release) to > provision HPC multi-interface compute nodes with stateless images. > > Basically, the nodes have one 1Gb/s nic and one 10Gb/s nic, each one > connected to a different switch. Obvioulsy, I want them to be > provisioned through the 10Gb/s link. > > I'm using switch-based discovery and, provided that BIOS PXE priorities > is correctly configured, everything works as expected. > But I was wondering what would happen if BIOS was wrongly configured to > PXE on the 1Gb/s nic first : would discover happen ? if yes, would it > result to ? > I first though it would be similar to a serial discovery situation but > it doesn't seem that simple, here's why : > > The compute nodes are physical nodes equipped with > > - 2 onboard 1Gb/s nic (eth0, eth1) > - 1 additionnal PCI-E 10Gb/s nic (eth2) > > and set up like this : > > - eth2 is connected to a 10Gb/s ports switch which is used by > switch-based discovery ('switch' and 'switchport' attributes point to > this switch and switch ports) > > - eth0 is connected to a 1Gb/s ports switch and is used for ipmi traffic > from/to the BMC and could also carry the data traffic (legacy from the > time we didn't have a 10Gb/s switch nor nic") > > - BIOS is set up to PXE boot first and PXE is set up to boot through > eth2 first. > > Here's what I tested : > > Starting conditions : > > - Let's say nodeA was discovered and netbooted through the method above > (thus its mac in the mac table is eth2 (10Gb/s) address). > > - let's say I haven't any undiscovered nor unprovisionned node > > To see what would happen through eth0, I changed the eth2 facing switch > configuration to forbid traffic coming from this interface just to be > sure eth2 PXE could no work no matter what. So now the node does PXE on > eth0 (1Gb/s) > > I know about the 3 types of automatic discovery as described in > documentation (MTMS, switch-based, serial) and here's what I tested : > > > 1) nodeA : > > - remove its mac address in the mac table > - remove all its leases in /var/lib/dhcpd/dhcpd.leases > - makedhcp -n > > reboot nodeA > > -> it is discovered and provisionned with its osimage but through the > eth0 interface. hostname is nodeA as expected > > in the mac table we can see > > "nodeA",,"<eth0 mac-address>|<eth2 mac-address>!*NOIP*",, > > -> seems strange to me (see at the end of my tests description) > > 2) nodeA > > - rmdef nodeA > - remove all its leases in /var/lib/dhcpd/dhcpd.leases > - makedhcp -n > - add a new node with simmilar properties (same chain, ...) : > > nodeadd foobar groups=<same as nodeA> > tabedit hosts : insert a specific entry with an ip for foobar > tabedit ipmi : insert a specific entry with an bmc ip for foobar > makedns foobar > > reboot nodeA > > -> it gets stalled ("Network configuration complete, commencing transmit > of discovery packets") and PXE loops after getting an IP from the dhcpd > dynamic range > > -> seems pretty normal to me since I consider myself to be in a serial > discovery like scenario then > > 3) same as 2) but with a nodediscoverstart noderange=foobar AFTER node > reboot : seems to loop in PXE/discovery (even if nodediscoverstop is run) > > -> seems to me it might be normal as I maybe was supposed to run > nodediscoverstart before node boot. > > 4) same as 2) but with a nodediscoverstart noderange=foobar BEFORE node > reboot > > -> foobar (same physical node as nodeA) gets discovered and > provisionned. hostname is foobar as expected > > in the mac table we can see > > "foobar",,"0c:c4:7a:4d:85:a8",, > > -> seems pretty normal to me > > [I don't quite understand what is happening in 3) but it a bit off topic] > > So, basically, I don't understand the conceptual difference between 1) > and 4) and thus why in 1) nodeA gets discovered without the need to run > nodediscoverstart (granted it has switch and swithport attributes but > not refering the 1Gb/s switch)... > > Again, I'd like to understand the above in order to be sure that the > node won't get provisionned through the eth0/1Gb/s link if PXE order > gets mixed up...Or even to make sure (worst case scenario if many nodes > would have PXE order mixed up) that nodes hostnames won't be mixed up > (in a serial discovery like scenario). > > Thanks > > -- > Thomas HUMMEL > > > _______________________________________________ > xCAT-user mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/xcat-user >
_______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
