Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
Here's what's in our /etc/modprobe.conf related to IB and lustre: options ib_mthca msi_x=1 options lnet networks=o2ib0(ib0) options ko2iblnd ipif_name=ib0 We have Mellanox Infinihost (III?) DDR cards and IPs defined for them. $ /sbin/ifconfig ib0 ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE: 80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask: 255.255.255.0 inet6 addr: fe80::202:c902:29:b341/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:1719 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:96564 (94.3 KiB) TX bytes:2420 (2.3 KiB) Albert On Feb 20, 2011, at 2:20 PM, Arya Mazaheri wrote: I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/ modprobe.conf ? On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett aeever...@ualr.edu wrote: For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. / etc/modprobe.conf needs a tweak. For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware. Albert On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote: Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/ modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
Thanks Albert, I really appreciate you... Now everything is working... On Mon, Feb 21, 2011 at 7:44 PM, Albert Everett aeever...@ualr.edu wrote: Here's what's in our /etc/modprobe.conf related to IB and lustre: options ib_mthca msi_x=1 options lnet networks=o2ib0(ib0) options ko2iblnd ipif_name=ib0 We have Mellanox Infinihost (III?) DDR cards and IPs defined for them. $ /sbin/ifconfig ib0 ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:29:b341/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:1719 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:96564 (94.3 KiB) TX bytes:2420 (2.3 KiB) Albert On Feb 20, 2011, at 2:20 PM, Arya Mazaheri wrote: I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/modprobe.conf ? On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett aeever...@ualr.edu wrote: For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/modprobe.conf needs a tweak. For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware. Albert On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote: Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
Excellent - thanks for letting us know. Thanks again to Tim Carlson for his OFED wiki, and to Terascala for help with building up the lustre client. Albert On Feb 21, 2011, at 12:09 PM, Arya Mazaheri wrote: Thanks Albert, I really appreciate you... Now everything is working... On Mon, Feb 21, 2011 at 7:44 PM, Albert Everett aeever...@ualr.edu wrote: Here's what's in our /etc/modprobe.conf related to IB and lustre: options ib_mthca msi_x=1 options lnet networks=o2ib0(ib0) options ko2iblnd ipif_name=ib0 We have Mellanox Infinihost (III?) DDR cards and IPs defined for them. $ /sbin/ifconfig ib0 ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE: 80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask: 255.255.255.0 inet6 addr: fe80::202:c902:29:b341/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:1719 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:96564 (94.3 KiB) TX bytes:2420 (2.3 KiB) Albert On Feb 20, 2011, at 2:20 PM, Arya Mazaheri wrote: I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/ modprobe.conf ? On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett aeever...@ualr.edu wrote: For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. / etc/modprobe.conf needs a tweak. For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware. Albert On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote: Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/ modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/modprobe.conf ? On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett aeever...@ualr.edu wrote: For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/modprobe.conf needs a tweak. For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware. Albert On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote: Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
On 11-02-19 09:34 AM, Arya Mazaheri wrote: As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. You need the client modules package that matches your kernel. If one is not available you will have to build it from the source. b. signature.asc Description: OpenPGP digital signature ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
Well, I want to install client module on rocks 5.5 x86_64. there are some packages in source section of lustre download area. I am confused which one to choose? what are the differences between them? lustre-client-source-2.0.0.1-2.6.18_164.11.1.0.1.el5_lustre.2.0.0.1.i686.rpm lustre-client-source-2.0.0.1-2.6.16_60_0.42.8_lustre.2.0.0.1_smp.x86_64.rpm lustre-client-source-2.0.0.1-2.6.27_23_0.1_lustre.2.0.0.1_default.x86_64.rpm lustre-client-source-2.0.0.1-2.6.18_164.11.1.0.1.el5_lustre.2.0.0.1.x86_64.rpm lustre-client-source-2.0.0.1-2.6.16_60_0.42.8_lustre.2.0.0.1_bigsmp.i686.rpm lustre-client-source-2.0.0.1-2.6.27_23_0.1_lustre.2.0.0.1_default.i686.rpm lustre-client-source-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.i686.rpm lustre-client-source-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm and one other thing! where is the source for lustre-client-modules? On Sat, Feb 19, 2011 at 6:08 PM, Brian J. Murrell br...@whamcloud.comwrote: On 11-02-19 09:34 AM, Arya Mazaheri wrote: As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. You need the client modules package that matches your kernel. If one is not available you will have to build it from the source. b. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel
For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/ modprobe.conf needs a tweak. For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware. Albert On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote: Hi Albert, It seems that you have made a new kernel in order to run lustre on clients. Am I right? I don't want to change kernel on clients at all... On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett aeever...@ualr.edu wrote: Our kernel is also 2.6.18_194.17.4.el5. We installed OFED 1.5.2 from source, following this guide: https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster ... which left us, among other things, a folder /usr/src/ofa_kernel. Lustre on the server side is handled by our vendor, so all we needed to worry about is the client. To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below. # ./configure --with-linux=/lib/modules/`uname -r`/build # make rpms To get the lustre installation to use our new OFED, we tried this and it worked. # ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/ modules/`uname -r`/build # make rpms RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines. Albert On Feb 19, 2011, at 8:34 AM, Arya Mazaheri wrote: Hi, I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error: $ mount -t lustre 192.168.0.1:/lustre /mnt/lustre mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory: /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem. Any ideas? Thanks ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
