Re: [Lustre-discuss] Seeing OST errors on the OSS that doesnt have it mounted
On Aug 26, 2008 22:56 -0700, Alex Lee wrote: > I'm getting these error messages on my OSS. However the OST006 is not mounted > on this OSS(lustre-oss-0-0). OST0006 is visible to the server because > lustre-oss-0-0 is the failover node for lustre-oss-0-1 which does mount > OST0006. > > Should I be worried about these errors? I dont understand why the OSS is even > giving these errors out since there is no hardware issue that I can see. Also > the OST is not mounted on that OSS I would think its "invisible" to the OSS. > > I only get these errors during lustre usage. When the filesystem is not used > I never get any errors. When a client has a problem accessing a service (OST or MDT) on the primary node (e.g. RPC timeout) it will retry on the same node first, then try the backup and continue to try both until one of them answers... > Aug 23 12:27:52 lustre-oss-0-0 kernel: LustreError: > 2918:0:(ldlm_lib.c:1536:target_send_reply_msg()) @@ > @ processing error (-19) [EMAIL PROTECTED] x52/t0 o8->@:0/0 lens 240/0 > e 0 to 0 dl 1219462372 > ref 1 fl Interpret:/0/0 rc -19/0 The fact that lustre-oss-0-0 returns -ENODEV (-19) isn't a reason to stop trying there, because it may take some time for OST to failover from primary server to backup. What this really means is that your primary server is having network trouble, or is so severely overloaded that the client has given up on it and is trying the backup. It could also be a problem on the client I guess. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] manual ost failover problems
On Aug 21, 2008 18:29 +1000, Marcus Schull wrote: > We are currently testing lustre 1.6.5.1 on RHEL 5 (64bit) with 3 OSTs > for a 'data' filesystem running on server1 and 4 OSTs for a 'common' > filesytem running on server2. Each OST is a 1TB SAN LUN that can be > seen from either server. The idea was to run the servers as an > active/active failover pair, being able to mount the 'other' LUNs on > the remaining server if one server failed. Also, we could have the > flexibility of striping (between the 2 nodes initially --> more in > the future), if the OSTs of each fs were spread out amongst the > servers. > > At present, this works well if all LUNs are only mounted on the > initial server they are mounted on after creation. > > I had assumed that OSTs could be unmounted from server1 and then > remounted on then remounted on server2 (never simultaneously > mounted), but this does not seem to work whether or not clients are > using (have mounted) the file system, or even whether the servers are > rebooted in between the change. > > Even though the LUNs will mount on the other server, any clients that > access the filesytem will 'hang' until the LUN is mounted back in its > initial location. > > The filesystems were created using the --failnode option. Odd, this is exactly what should work. Do the clients report trying to contact the backup server? > Is there a command to 'update' the ?MGS/MDT's information regarding > this, and so communicate this to the clients? No, the clients should know this from the configuration they got at mount time, and try automatically with the backup server if the primary is down. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] OST failover
Hi the list, I have 4 OSS servers , each OSS has 1 gigabit network interface and 2 OSTs, first OST use /dev/sdb1, second OST use /dev/sdb2. OSS1: OST1 & OST2 OSS2: OST3 & OST4 OSS3: OST5 & OST6 OSS4: OST7 & OST8 Each file is striped across 4 OSTs : OST1,2,3,4 And I want to setup failover as: OST2 failover with OST3 OST4 failover with OST5 OST6 failover with OST7 OST8 failover with OST1 How do I do? Thanks. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] lustre patches for e2fsprogs version 1.41.0?
Hello, Are there any plans to support the latest e2fsprogs version with the lustre patches? If yes, is there a a bug in the bugzilla where the status can be viewed? Greetings Patrick Winnertz ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] OST failover
Hi, You interconnect OSS1 and OSS2, OSS2 and OSS3, OSS3 and OSS4, and OSS4 and OSS1 together via an out-of-band communication fabric, install Linux-HA heartbeat software to monitor mount points, and configure nodes to take over each other¹s mounts in case of failure. The implementation I have here uses only single point-to-point links between OSS pairs (there is no crossover between failover partners) but the princple is the same; your implementation should work pretty much the same way, with a few minor differences. I¹m assuming you¹ve got unused NICs on your OSSes, so if you connect those to a standard GigE edge switch protected by UPS, you can create an HA fabric which will give you the coverage you need. Lustre itself doesn¹t support failover natively, you need to implement any failover using a third-party package; most of us out there use Linux-HA to great effect. cheers, Klaus On 8/27/08 2:30 AM, "sd a" <[EMAIL PROTECTED]>did etch on stone tablets: > Hi the list, > > I have 4 OSS servers , each OSS has 1 gigabit network interface and 2 OSTs, > first OST use /dev/sdb1, second OST use /dev/sdb2. > > OSS1: OST1 & OST2 > > OSS2: OST3 & OST4 > > OSS3: OST5 & OST6 > > OSS4: OST7 & OST8 > > > Each file is striped across 4 OSTs : OST1,2,3,4 > > And I want to setup failover as: > > OST2 failover with OST3 > > OST4 failover with OST5 > > OST6 failover with OST7 > > OST8 failover with OST1 > > > How do I do? > > Thanks. > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Lustre Patchless Client
Hello, I'm new to lustre community and lustre software as well. I have question regarding to patchless lustre client installation. The OS is Redhat EL 5.1. I'm using voltaire gridstack v5.1.3 for infiniband software stack. I installed the lustre-1.6.5.1 servers without any problem. The client and server OSes are same. The problem is installing patchless client. Are there any howto or guide for that. I googled but got nothing about it. Just the link http://wiki.lustre.org/index.php?title=Patchless_Client . When i tried the things which i read on that page, no modules were build. Are there anything that must be done for the patchless client installation. I ran the following command for configuration: ./configure --prefix=/usr/local/lustre --enable-uoss --enable-bgl --enable-posix-osd --enable-panic_dumplog --enable-quota --enable-health-write --enable-lru-resize --enable-adaptive-timeouts --enable-efence --enable-libwrap --enable-snmp --with-o2ib=/usr/src/openib --with-linux=/usr/src/kernels/2.6.18-53.1.14.el5 Then ran make && make install. None of the above steps returned error messages. So I assumed that the compilation and installation is successful. But when i try to mount the server the following error message returned: mount.lustre: mount [EMAIL PROTECTED]:/klfs at /klfs failed: No such device Are the lustre modules loaded? Check /etc/modprobe.conf and /proc/filesystems Note 'alias lustre llite' should be removed from modprobe.conf I searched any module files that contains lnet in its name. But there were no files. So the compilation and installation is not complete. And also i installed lustre-client-modules-1.6.5.1-2.6.18_53.1.14.el5_lustre.1.6.5.1smp.x86_64.rpm package to have the modules. But (as I expected,) it also did not help. After installing that package, the lnet modules are installed under the /lib/modules/2.6.18-53.1.14.el5/kernel/net/lustre dir. But it did not help. When i ran the modprobe lnet command, it returned me "FATAL: Module lnet not found." message. And ran the insmod command as below: insmod /lib/modules/2.6.18-53.1.14.el5/kernel/net/lustre/lnet.ko insmod: error inserting '/lib/modules/2.6.18-53.1.14.el5/kernel/net/lustre/lnet.ko': -1 Unknown symbol in module All the trial and error steps made me believe that there must be other things to do for installation. Are there any step by step guide for client installation? Could you please help me? And one more question: Are there any possibility to run two different lustre versions on the same client at the same time? If there is a way to do it, it would be great... Thanks in advance. Best Regards, Ender GULER System Administrator UYBHM [http://www.uybhm.itu.edu.tr] ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre patches for e2fsprogs version 1.41.0?
On Aug 27, 2008 12:43 +0200, Patrick Winnertz wrote: > Are there any plans to support the latest e2fsprogs version with the lustre > patches? If yes, is there a a bug in the bugzilla where the status can be > viewed? Yes, we update e2fsprogs to follow upstream fairly regularly, and will be making a release based on 1.41 fairly soon. Is there something specific you are looking for from the 1.41 release? The largest difference is that 1.41 has upstream support for the Lustre extents format (also used in ext4), but the code is completely different. It would be interesting to see whether the upstream e2fsck can fix all of the test cases we have for corrupted extents. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Lustre backups using Amanda
Amanda (http://amanda.zmanda.com) is an open source network backup and recovery software. I have added a white paper to the lustre wiki (http://wiki.lustre.org/index.php?title=Main_Page#Community_Development_Projects). Look for "Backup: Amanda and Lustre". This paper discusses how to use Amanda to backup Lustre file systems. If you have comments, please send them to me or discuss it in this list. thanks, Paddy -- Amanda http://amanda.zmanda.com ZRM for MySQL http://www.zmanda.com/backup-mysql.html ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Softlockup issues. Lustre related?
Hello Folks, I have few client nodes that are getting soft lockup errors. These are patchless clients running Lustre 1.6.5.1 with kernel 2.6.18-53.1.6.el5-PAPI. More or less stock RHEL 5.1 with PAPI patch added on it. The MDS and OSS are running Lustre 1.6.5.1 with the supplied Lustre kernels and OFED 1.3.1. I remember there was an issue with __d_lookup in the past but I thought it was fixed with the newest release of Lustre. So I dont know if this is related in anyway at all. I dont see any other real lustre error messages on the client or the MDS/OSS at the time of the softlock up. Also wasnt there a softirq issue? I dont think this is related to that... Thanks, -Alex Here is the syslog output: Aug 24 04:02:03 papi-0476 syslogd 1.4.1: restart. Aug 25 00:52:24 papi-0476 kernel: Losing some ticks... checking if CPU frequency changed. Aug 25 15:17:05 papi-0476 kernel: cpsMPI.x[10493]: segfault at 2aaab23ce000 rip 004a8fce rs p 7fff584d6cf0 error 4 Aug 27 04:15:00 papi-0476 kernel: BUG: soft lockup detected on CPU#4! Aug 27 04:15:00 papi-0476 kernel: Aug 27 04:15:00 papi-0476 kernel: Call Trace: Aug 27 04:15:00 papi-0476 kernel:[] softlockup_tick+0xd5/0xe7 Aug 27 04:15:00 papi-0476 kernel: [] tasklet_action+0x62/0xac Aug 27 04:15:00 papi-0476 kernel: [] update_process_times+0x54/0x7a Aug 27 04:15:00 papi-0476 kernel: [] smp_local_timer_interrupt+0x2c/0x61 Aug 27 04:15:00 papi-0476 kernel: [] call_softirq+0x1c/0x28 Aug 27 04:15:00 papi-0476 kernel: [] smp_apic_timer_interrupt+0x41/0x47 Aug 27 04:15:00 papi-0476 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 27 04:15:00 papi-0476 kernel:[] dummy_inode_permission+0x0/0x3 Aug 27 04:15:00 papi-0476 kernel: [] __d_lookup+0xd2/0xff Aug 27 04:15:00 papi-0476 kernel: [] __d_lookup+0xb0/0xff Aug 27 04:15:00 papi-0476 kernel: [] do_lookup+0x2c/0x1d4 Aug 27 04:15:00 papi-0476 kernel: [] __link_path_walk+0xa01/0xf42 Aug 27 04:15:00 papi-0476 kernel: [] link_path_walk+0x5c/0xe5 Aug 27 04:15:00 papi-0476 kernel: [] do_path_lookup+0x270/0x2e8 Aug 27 04:15:01 papi-0476 kernel: [] __user_walk_fd+0x37/0x4c Aug 27 04:15:01 papi-0476 kernel: [] vfs_lstat_fd+0x18/0x47 Aug 27 04:15:01 papi-0476 kernel: [] sys_newfstatat+0x22/0x43 Aug 27 04:15:01 papi-0476 kernel: [] system_call+0x7e/0x83 Aug 27 04:15:01 papi-0476 kernel: Aug 28 04:03:14 papi-0476 kernel: BUG: soft lockup detected on CPU#0! Aug 28 04:03:14 papi-0476 kernel: Aug 28 04:03:14 papi-0476 kernel: Call Trace: Aug 28 04:03:14 papi-0476 kernel:[] softlockup_tick+0xd5/0xe7 Aug 28 04:03:14 papi-0476 kernel: [] update_process_times+0x54/0x7a Aug 28 04:03:14 papi-0476 kernel: [] smp_local_timer_interrupt+0x2c/0x61 Aug 28 04:03:14 papi-0476 kernel: [] smp_apic_timer_interrupt+0x41/0x47 Aug 28 04:03:14 papi-0476 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 28 04:03:14 papi-0476 kernel:[] dummy_inode_permission+0x0/0x3 Aug 28 04:03:14 papi-0476 kernel: [] __d_lookup+0xe2/0xff Aug 28 04:03:14 papi-0476 kernel: [] __d_lookup+0xb0/0xff Aug 28 04:03:14 papi-0476 kernel: [] do_lookup+0x2c/0x1d4 Aug 28 04:03:14 papi-0476 kernel: [] __link_path_walk+0xa01/0xf42 Aug 28 04:03:14 papi-0476 kernel: [] link_path_walk+0x5c/0xe5 Aug 28 04:03:14 papi-0476 kernel: [] do_path_lookup+0x270/0x2e8 Aug 28 04:03:14 papi-0476 kernel: [] __user_walk_fd+0x37/0x4c Aug 28 04:03:14 papi-0476 kernel: [] vfs_lstat_fd+0x18/0x47 Aug 28 04:03:14 papi-0476 kernel: [] sys_newfstatat+0x22/0x43 Aug 28 04:03:14 papi-0476 kernel: [] system_call+0x7e/0x83 Aug 28 04:03:14 papi-0476 kernel: Aug 28 04:12:12 papi-0476 kernel: BUG: soft lockup detected on CPU#3! Aug 28 04:12:17 papi-0476 kernel: Aug 28 04:12:18 papi-0476 kernel: Call Trace: Aug 28 04:12:18 papi-0476 kernel:[] softlockup_tick+0xd5/0xe7 Aug 28 04:12:18 papi-0476 kernel: [] update_process_times+0x54/0x7a Aug 28 04:12:18 papi-0476 kernel: [] smp_local_timer_interrupt+0x2c/0x61 Aug 28 04:12:18 papi-0476 kernel: [] smp_apic_timer_interrupt+0x41/0x47 Aug 28 04:12:18 papi-0476 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 28 04:12:18 papi-0476 kernel: [] kmem_cache_free+0x1c0/0x1cb Aug 28 04:12:18 papi-0476 kernel: [] __rcu_process_callbacks+0x122/0x1a8 Aug 28 04:12:18 papi-0476 kernel: [] rcu_process_callbacks+0x23/0x43 Aug 28 04:12:18 papi-0476 kernel: [] tasklet_action+0x62/0xac Aug 28 04:12:18 papi-0476 kernel: [] __do_softirq+0x5e/0xd5 Aug 28 04:12:18 papi-0476 kernel: [] call_softirq+0x1c/0x28 Aug 28 04:12:18 papi-0476 kernel: [] do_softirq+0x2c/0x85 Aug 28 04:12:18 papi-0476 kernel: [] apic_timer_interrupt+0x66/0x6c Aug 28 04:03:14 papi-0476 kernel: [] sys_newfstatat+0x22/0x43 Aug 28 04:03:14 papi-0476 kernel: [] system_call+0x7e/0x83 Aug 28 04:03:14 papi-0476 kernel: Aug 28 04:12:12 papi-0476 kernel: BUG: soft lockup detected on CPU#3! Aug 28 04:12:17 papi-0476 kernel: Aug 28 04:12:18 papi-0476 kernel: Call Trace: Aug 28 04:12:18 papi-0476 kernel:[] softlockup_tick+0xd5/0xe7 Aug 28 04:
Re: [Lustre-discuss] Seeing OST errors on the OSS that doesnt have it mounted
Andreas Dilger wrote: > On Aug 26, 2008 22:56 -0700, Alex Lee wrote: > >> I'm getting these error messages on my OSS. However the OST006 is not >> mounted on this OSS(lustre-oss-0-0). OST0006 is visible to the server >> because lustre-oss-0-0 is the failover node for lustre-oss-0-1 which does >> mount OST0006. >> >> Should I be worried about these errors? I dont understand why the OSS is >> even giving these errors out since there is no hardware issue that I can >> see. Also the OST is not mounted on that OSS I would think its "invisible" >> to the OSS. >> >> I only get these errors during lustre usage. When the filesystem is not used >> I never get any errors. >> > > When a client has a problem accessing a service (OST or MDT) on the primary > node (e.g. RPC timeout) it will retry on the same node first, then try the > backup and continue to try both until one of them answers... > > >> Aug 23 12:27:52 lustre-oss-0-0 kernel: LustreError: >> 2918:0:(ldlm_lib.c:1536:target_send_reply_msg()) @@ >> @ processing error (-19) [EMAIL PROTECTED] x52/t0 o8->@:0/0 lens >> 240/0 e 0 to 0 dl 1219462372 >> ref 1 fl Interpret:/0/0 rc -19/0 >> > > The fact that lustre-oss-0-0 returns -ENODEV (-19) isn't a reason to stop > trying there, because it may take some time for OST to failover from primary > server to backup. > > What this really means is that your primary server is having network > trouble, or is so severely overloaded that the client has given up on > it and is trying the backup. It could also be a problem on the client > I guess. > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > Andreas, Thanks for the info. Is there any documentation on how to decode the error messages? I feel bad keep posting on the list for every single error message I dont understand. Thanks, -Alex ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] csum errors
We recently upgraded from 1.4.10.1 to 1.6.5.1 (clients and servers) and now we are seeing errors like Aug 27 07:49:54 oss025 kernel: LustreError: 3738:0:(ost_handler.c: 1163:ost_brw_write()) client csum 2dbc1696, server csum 9d081697 Aug 27 07:49:54 oss025 kernel: LustreError: 168-f: p1-OST0018: BAD WRITE CHECKSUM: changed in transit before arrival at OST from [EMAIL PROTECTED] inum 24522277/426969871 object 12021/0 extent [10485760-11534335] Aug 27 07:49:55 oss025 kernel: LustreError: 3738:0:(ost_handler.c: 1225:ost_brw_write()) client csum 2dbc1696, original server csum 9d081697, server csum now 9d081697 always from the same cluster node... Should we be worried? I suspect this means we shouldn't turn check summing off? I assume these are rejected and resent from the client? -- Dr Stuart Midgley [EMAIL PROTECTED] ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
