Thanks to David Johnson from brown.edu who followed up on my message. I forgot
that when I copied over the boot image from cluster 1 to cluster 2 I needed to
rerun nodeset, so that the initrd that had the updated bfa.ko in it was copied
from /install/netboot and placed into /tftpboot. Thanks again Dave!
-Brad Viviano
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - [email protected]
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - [email protected]
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - [email protected]
________________________________
From: Viviano, Brad <[email protected]>
Sent: Thursday, March 6, 2014 10:58 AM
To: [email protected]
Subject: [xcat-user] Wrong kernel module getting loaded in stateless image.
Hello,
I have 2 iDataplex clusters both running RHEL6.4, based on November 2013
best recipe. Both clusters are using IBM x3650M3 nodes for GPFS servers. The
nodes are stateless and boot from an image that is shared between the two
clusters. The x3650M3's in both clusters all have Brocade 825 fiber channel
cards, attached to backend DS storage.
All the 825 cards are running firmware 3.2.3.0, inline with the November
2013 best recipe. I also loaded the 3.2.3.0 brocade drivers into the diskless
image. Cluster 1 GPFS nodes are working fine, cluster 2 GPFS nodes, which I am
now setting up, are not. For some reason, the GPFS nodes of cluster 2 are
loading the RHEL6.4 stock bfa.ko driver. Thing is, both clusters are using the
same boot image and initrd.
It took me some effort back in January when I setup cluster 1 to get all
the Brocade firmware and drivers loaded and correct. I did everything I did on
cluster 1 for cluster 2, but I can't seem to get cluster2's GPFS server nodes
to load the updated bfa.ko.
>From the GPFS storage image:
# find /lib/modules/2.6.32-358.el6.x86_64 -name bfa.ko
/lib/modules/2.6.32-358.el6.x86_64/extra/bfa/bfa.ko
/lib/modules/2.6.32-358.el6.x86_64/kernel/drivers/scsi/bfa/bfa.ko
kernel/drivers/scsi/bfa/bfa.ko is the RHEL6.4 stock driver, version 3.0.23.0
extra/bfa/bfa.ko is the updated RHEL6.4 driver from IBM, version 3.2.3.0
I know it's loading the older kernel module on cluster2 nodes because if I try
and use the "bcu" utility from Brocade, it fails:
# bcu --version
Brocade CLI utility
Version:3.2.3.0 09/06/2013 00.34.46
# bcu adapter --list
ERROR: Brocade adapter/driver not installed or loaded
If I use the older 3.0.23.0 utility, it works:
# bcu-old --version
Brocade CLI utility
Version:3.0.23.0 11/08/2012 10.08.51
# bcu-old adapter --list
------------------------------------------------------------------------
AD# NP HW-path Type Model-Info Serial-num Name
------------------------------------------------------------------------
1 2 0000:15:00 HBA Brocade-825 AVM0443E03V --
2 2 0000:1f:00 HBA Brocade-825 AVM0443E05X --
------------------------------------------------------------------------
If I rmmod bfa.ko and modprobe it, it gets the new driver and the newer bcu
utility works.
# lsmod | grep bfa
bfa 524835 0
# rmmod bfa
# modprobe bfa
# lsmod | grep bfa
bfa 2330780 0
You can see the size difference between the 3.0.23.0 and 3.2.3.0. and then the
bcu utility works fine:
# bcu adapter --list
------------------------------------------------------------------------
AD# NP HW-path Type Model-Info Serial-num Name
------------------------------------------------------------------------
1 2 0000:15 HBA Brocade-825 AVM0443E03V --
2 2 0000:1f HBA Brocade-825 AVM0443E05X --
------------------------------------------------------------------------
Since I am using the same image between the two clusters, there must be
something different in my xCAT setup that is causing the newer module to get
loaded in Cluster 1 that I am missing for Cluster 2, but I can't see what it
would be. I've done a side-by-side comparison of lsdef for the two nodes and
other then MAC address and IP, they are the same. I thought I had documented
all the steps when I built cluster 1, but apparently I am missing something.
Any help would be appreciated.
Thanks,
-Brad Viviano
===================================================
Brad Viviano
High Performance Computing & Scientific Visualization
Lockheed Martin, Supporting the EPA
Research Triangle Park, NC
919-541-2696
HSCSS Task Order Lead - Ravi Nair
919-541-5467 - [email protected]
High Performance Computing Subtask Lead - Durward Jones
919-541-5043 - [email protected]
Environmental Modeling and Visualization Lead - Heidi Paulsen
919-541-1834 - [email protected]
------------------------------------------------------------------------------
Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce.
With Perforce, you get hassle-free workflows. Merge that actually works.
Faster operations. Version large binaries. Built-in WAN optimization and the
freedom to use Git, Perforce or both. Make the move to Perforce.
http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
