Hello,
I’ve been working to set up a small cluster with xCAT using switch-based
discovery, and everything was going perfectly with the first switch, but now
I’m running into issues where nodes on a second switch are clobbering the
previous ones. I can’t seem to track down where this is happening.
Setup:
Nodes hypnonode[001-028] connected to switch eth-c33-1, ports 1-28
Nodes hypnonode[029-042] connected to switch eth-d33-1, ports 1-14
Running xCAT 2.13.4
# lsxcatd -v
Version 2.13.4 (git commit 6ee3741498768994e4bb10d2a77c9699bcabde90, built Tue
May 16 10:03:13 EDT 2017)
The nodes have the correct switch/switchport definitions:
# lsdef hypnonode[001,029] -i switch,switchport
Object name: hypnonode001
switch=eth-c33-1
switchport=1
Object name: hypnonode029
switch=eth-d33-1
switchport=1
SNMP is configured properly and the settings are correct in the switches table:
# snmpget -v3 -a sha -u XXXXX -A XXXXX -l authNoPriv eth-c33-1
SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: eth-c33-1
# snmpget -v3 -a sha -u XXXXX -A XXXXX -l authNoPriv eth-d33-1
SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: eth-d33-1
# tabdump switches
#switch,snmpversion,username,password,privacy,auth,linkports,sshusername,sshpassword,protocol,switchtype,comments,disable
"eth-c33-1","3","XXXXX","XXXXX",,"sha",,,,,,,
"eth-d33-1","3","XXXXX","XXXXX",,"sha",,,,,,,
Also, nodes 001-028 show up properly in the output of nodediscoverls, e.g.:
UUID NODE METHOD
MTM SERIAL
4C4C4544-0035-3710-8037-B8C04F424B31 hypnonode023 switch
DELL: 8577BK1
4C4C4544-0051-3510-8058-C7C04F4B4D31 hypnonode001 switch
DELL: GQ5XKM1
[…]
However, when I boot up a node connected to eth-d33-1, it gets assigned to be
hypnonode001. I’ve noticed that this even happens irrespective of the switch
port, but rather in the order the machines are booted. I did confirm that
Sequential Discovery is *not* running:
# nodediscoverstatus
Sequential Discovery is stopped.
Profile Discovery is stopped.
Relevant bits from computes.log and cluster.log:
Jul 20 11:43:51 10.137.5.227 (none) rsyslogd: [origin software="rsyslogd"
swVersion="5.8.10" x-pid="1430" x-info="http://www.rsyslog.com";] start
Jul 20 11:43:51 10.137.5.227 (none) xcat.genesis.doxcat: Getting initial
certificate --> 10.137.4.8:3001
Jul 20 11:43:51 10.137.5.227 (none) dhclient[468]: XMT: Solicit on eth0,
interval 7810ms.
Jul 20 11:43:52 10.137.5.227 (none) xcat.genesis.doxcat: Running getdestiny
--> 10.137.4.8:3001
Jul 20 11:43:52 10.137.5.227 (none) xcat.genesis.doxcat: Received
destiny=discover
Jul 20 11:43:52 10.137.5.227 (none) xcat.genesis.doxcat: The destiny=discover,
destiny parameters=discover
Jul 20 11:43:52 10.137.5.227 (none) xcat.genesis.doxcat: Running dodiscovery...
Jul 20 11:43:52 10.137.5.227 (none) xcat.genesis.dodiscovery: Beginning node
discovery process...
Jul 20 11:43:59 10.137.5.227 (none) dhclient[468]: XMT: Solicit on eth0,
interval 16300ms.
Jul 20 11:44:02 10.137.5.227 (none) xcat.genesis.dodiscovery: No link detected
on eth1#012 eth2#012 eth3 after 10 seconds
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Waiting for nics
to get addresses
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Network
configuration complete, commencing transmit of discovery packets
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Beginning echo
infomation to discovery packet file...
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Discovery packet
file is ready.
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Sending the
discovery packet to xCAT (10.137.4.8:3001)...
Jul 20 11:44:03 10.137.5.227 (none) xcat.genesis.dodiscovery: Sleeping 5
seconds...
Jul 20 11:44:03 hypnotoad-mgt01 xcat[13593]: xcatd: Processing discovery
request from 10.137.5.227
Jul 20 11:44:03 hypnotoad-mgt01 xcat[13593]: xcat.discovery.aaadiscovery:
(00:21:9b:a2:57:b1) Got a discovery request, attempting to discover the node...
Jul 20 11:44:03 hypnotoad-mgt01 xcat[13593]: xcat.discovery.blade:
(00:21:9b:a2:57:b1) Warning: Could not find any nodes using blade-based
discovery
Jul 20 11:44:04 hypnotoad-mgt01 xcat[13593]: xcat.discovery.switch:
(00:21:9b:a2:57:b1) Found node: hypnonode001
Jul 20 11:44:06 hypnotoad-mgt01 xcat[13593]: xcat.discovery.nodediscover:
hypnonode001 has been discovered
Jul 20 11:44:06 hypnotoad-mgt01 xcat[13593]: xcat.discovery.zzzdiscovery:
(00:21:9b:a2:57:b1) Successfully discovered the node using switch discovery
method.
Jul 20 11:44:08 10.137.5.227 (none) xcat.genesis.dodiscovery: Restart network
interfaces...
Jul 20 11:44:08 10.137.5.227 (none) dhclient[2709]: DHCPRELEASE on eth0 to
10.137.4.8 port 67 (xid=0x62d52f2b)
Jul 20 11:44:13 hypnonode001 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.10" x-pid="2784" x-info="http://www.rsyslog.com";] start
Jul 20 11:44:13 hypnonode001 xcat.genesis.doxcat: Beginning doxcat process...
Jul 20 11:44:13 hypnonode001 xcat.genesis.doxcat: Waiting for device with
address 00:21:9b:a2:57:b1 to appear..
What I *don’t* understand, though, is that I took a look at the discopacket
file, and it clearly lists the correct switch there (abbreviated output):
<xcatrequest>
<uuid>4C4C4544-0051-4410-805A-C7C04F4B4D31</uuid>
<hwaddr>00:21:9B:A2:57:B1</hwaddr>
<switchname>eth-d33-1</switchname>
<switchaddr>10.137.5.248</switchaddr>
<switchport>1</switchport>
</xcatrequest>
At this point, nodediscoverls shows two both nodes as being hypnonode001:
# nodediscoverls | grep hypnonode001
4C4C4544-0051-3510-8058-C7C04F4B4D31 hypnonode001 switch
DELL: GQ5XKM1
4C4C4544-0051-4410-805A-C7C04F4B4D31 hypnonode001 switch
DELL:makena-2-3.localGQDZKM1
I’ve run out of ideas to check. Please let me know if you need any other
information to help track down why this is happening and/or how to fix it.
Thanks,
Tom
--
Tom Indelli
[email protected]<mailto:[email protected]>
Systems Administrator
The University of Chicago
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
