It does fix the problem thanks very much. I'll need to do something similar for the kvm stateless image too (where we are using SL 6.2 with patches).
Chris On 20/05/15 14:12, Jarrod Johnson wrote: > Just do 'mknb x86_64' after that change, it should help. > > Jarrod Johnson > HPC Systems Management Architect > Lenovo > > > -----Original Message----- > From: Christopher J. Walker [mailto:c.j.wal...@qmul.ac.uk] > Sent: Tuesday, May 19, 2015 5:24 PM > To: xcat-user@lists.sourceforge.net > Subject: Re: [xcat-user] xcat genesis kernel and mellanox ethernet cards > > On 19/05/15 19:50, Jarrod Johnson wrote: >> I am starting to guess that you have a VPI adapter that doesn't set the PCI >> class to Ethernet.. amending the 98-mlx.rules to remove the criteria for >> class may do the trick... >> > > Yes, I think we do have a VPI adapter (though it wasn't me that bought it, so > I'm not entirely sure). > > I attach the output of lspci and lspci -n , but I think that the pertinent > lines are: > > from lspci: > > 4b:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] > > lspci -n: > > 4b:00.0 0280: 15b3:1003 > > > So I need to change: > > [root@mgt.apocrita rules.d]# cat 98-mlx.rules SUBSYSTEM=="pci", > ATTRS{subsystem_vendor}=="0x15b3", > ATTRS{class}=="0x020000", RUN+="/sbin/loadmlxeth" > > to > SUBSYSTEM=="pci", ATTRS{subsystem_vendor}=="0x15b3", RUN+="/sbin/loadmlxeth" > > and rebuild the genesis kernel? > > Chris > > >> Jarrod Johnson >> HPC Systems Management Architect >> Lenovo >> >> >> -----Original Message----- >> From: Jarrod Johnson [mailto:jjohns...@lenovo.com] >> Sent: Tuesday, May 19, 2015 2:20 PM >> To: xCAT Users Mailing list >> Subject: Re: [xcat-user] xcat genesis kernel and mellanox ethernet >> cards >> >> Interesting, what is the output of lspci and lspci -n? >> For what it is worth, >> /opt/xcat/share/xcat/netboot/genesis/x86_64/fs/etc/udev/rules.d/98-mlx.rules >> is the mechanism that is supposed to load the mellanox Ethernet driver since >> mlx4_core instead of mlx4_en hooks the device. >> >> Jarrod Johnson >> HPC Systems Management Architect >> Lenovo >> >> >> -----Original Message----- >> From: Christopher J. Walker [mailto:c.j.wal...@qmul.ac.uk] >> Sent: Tuesday, May 19, 2015 1:50 PM >> To: xcat-user@lists.sourceforge.net >> Subject: [xcat-user] xcat genesis kernel and mellanox ethernet cards >> >> I seem to have run into: >> >> http://sourceforge.net/p/xcat/bugs/4578/ >> >> but for x86_64 - which I don't quite understand as that implies it has been >> fixed. >> >> If I plug the mellanox ConnectX-3 card in and try to do node discovery, the >> node hangs with the following prompt on the screen: >> >> Waiting for device with address f4:52:14:0b:15:e1 to appear >> >> If I unplug the network cable (and 40->10 adapter) from the card, then the >> genesis kernel boots. If I then plug the network in and do: >> >> [root@frontend5 ~]# lsmod | grep mlx >> mlx4_core 225805 0 >> [root@frontend5 ~]# ip link show >> >> Doesn't show the network card, however: >> >> however: >> [root@frontend5 ~]# modprobe mlx4_en >> 11: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000 >> link/ether f4:52:14:8a:ef:11 brd ff:ff:ff:ff:ff:ff >> >> Does. >> >> >> I think what's happening is that although the mlx4_core driver is being >> loaded, the mlx4_en isn't - and having pxe/uefi network booted from the >> card, it can't see it - hence the node discovery process hanging. >> >> Any thoughts about how I'd fix this? >> >> Chris >> >> ---------------------------------------------------------------------- >> -------- One dashboard for servers and applications across >> Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ >> applications Performance metrics, stats and reports that give you Actionable >> Insights Deep dive visibility with transaction tracing using APM Insight. >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> >> ---------------------------------------------------------------------- >> -------- One dashboard for servers and applications across >> Physical-Virtual-Cloud Widest out-of-the-box monitoring support with >> 50+ applications Performance metrics, stats and reports that give you >> Actionable Insights Deep dive visibility with transaction tracing >> using APM Insight. >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> >> ---------------------------------------------------------------------- >> -------- One dashboard for servers and applications across >> Physical-Virtual-Cloud Widest out-of-the-box monitoring support with >> 50+ applications Performance metrics, stats and reports that give you >> Actionable Insights Deep dive visibility with transaction tracing >> using APM Insight. >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user > ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
