Re: [Lustre-discuss] Future of LusterFS?
Taking a break from my current non-computer related work.. My guess based on your success is your gear is not so much cheap, as *cost effective high MTBF commodity parts*. If you go for the absolute bargain basement stuff, you'll have problems as individual components flake out. If you spend way too much money on high-end multi-redundant whizbangs, you generally get two things.. redundancy, which in my mind often only serves to make the eventual failure worse, and high-quality, long MTBF components. If you can get the high MTBF components without all the redudancy (and associated complexity nightmare), then you win. On Fri, Apr 23, 2010 at 05:42:30PM +0800, Stu Midgley wrote: We run lustre on cheap off the shelf gear. We have 4 generations of cheapish gear in a single 300TB lustre config (40 oss's) It has been running very very well for about 3.5 years now. Would lustre have issues if using cheap off the shell components or would people here think you need to have high end maskines with built in redundancy for everything? Troy Benjegerdes 'da hozer'ho...@hozed.org CTO, Freedom Fertilizer, Sustainable wind to NH3, t...@freedomfertilizer.com Benjegerdes Farms TerraCarbo biofuels The challenge in changing the world is not in having great ideas, it's in having stupid simple ideas, as those are the ones that cause change. Intellectual property is one of those great complicated ideas that intellectuals like to intellectualize over, lawyers like to bill too much over, and engineers like to overengineer. Meanwhile, it's the stupid simple ideas underfoot that create wealth. -- Troy, Mar 2010 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre support on suse 11 with PPC64
Because we have so little demand for ppc64 packages, we don't produce them for SDLC. This I understand, just rebuild from source. Note that ppc64 is supported for the Lustre client only. There is no server support on ppc64. Not that this is a big deal realistically, since it would be difficult to find PPC64 hardware that works well for a server, unless you roll your own embedded system... But why? The linux kernel has no problems providing an endian-clean and wordsize-clean environment to operate in. Are Sparc64 servers supported? ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] root on lustre and timeouts
On Wed, Apr 29, 2009 at 10:39:20AM -0400, Robin Humble wrote: we are (happily) using read-only root-on-Lustre in production with oneSIS, but have noticed something odd... if a root-on-Lustre client node has been up for more than 10 or 12hours then it survives an MDS failure/failover/reboot event(*), but if the client is newly rebooted and has been up for less than this time, then it doesn't successfully reconnect after an MDS event and the node is ~dead. by trial and error I've also found that if I rsync /lib64, /bin, and /sbin from Lustre to a root ramdisk, 'echo 3 /proc/sys/vm/drop_caches', and symlink the rest of dirs to Lustre then the node sails through MDS events. leaving out any one of the dirs/steps leads to a dead node. so it looks like the Lustre kernel's recovery process is somehow tied to userspace via apps in /bin and /sbin? Now that's interesting.. What distro are you using? I have been toying with the idea of modifiying the Debian initramfs-tools boot ramdisk to include bushbox and dropbear-ssh in order to debug these kind of root-network-filesystem bugs. In my case, I'm running AFS as the root filesystem, and I have the 'afsd' in the ramdisk that gets started at boot. I'm wondering if the Lustre binaries that are necessary could be placed in the initrd as well. It would be nice if various distros could work 'out of the box' with readonly network filesystems. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Patchless server
On Fri, Oct 10, 2008 at 02:23:27PM -0400, Brian J. Murrell wrote: On Fri, 2008-10-10 at 12:51 -0500, Troy Benjegerdes wrote: While I think I understand why you say this, it very easily can sound like a monopolistic tactic to sell more Sun hardware. Heh. I'm not sure I'm going to be able to say anything that will convince you otherwise. But to your points I will say... I appreciate the effort ;) Certainly it would be nice to support everyone's kernels, but we have limited resources and our customers have told us what kernels they want support for and that's what we support. You have to appreciate that Lustre development costs money to keep going and that money has to come from somewhere and currently it's coming from customers who want RHEL and SLES kernels. If there was a business case in supporting Debian/Ubuntu kernels, I think we'd be doing it. That said, we are proud that Lustre has been able to continue as an Open Source development project and as such are happy to see the community take up the packaging of Debian/Ubuntu packages in some of the Debian distributions for the community user base. In addition, IIRC there was an offer made on this list to include some amount of Debian/Ubuntu packaging foo in our official source repository should somebody want to contribute something. I don't think anyone has stepped up (yet). I am still hopeful. I went through the process of installing on Debian a month or two ago. It seems to work relatively well. http://wiki.lustre.org/index.php?title=Debian_Install All this effort in packaging and QA problems seems to kinda be something that would just go away with a patchless server though. Which I think leads back to having good documentation on what each patch in the set is for, and what issues it has in getting merged into upstream kernel.org. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] big problem: read-only fs
# grep lustre /proc/mounts /dev/*sdb */mnt/cubefs/ost-1 lustre ro 0 0 Why don't I see sdb1 in /proc? Also why do I see ro in /proc? 3. samba:~$ cat /proc/fs/lustre/lov/cubefs-clilov-8100330c4800/target_obd samba:~$ I have an other cluster, on that it shows the right values (on the same machine at the same time too). There is no answer about any of these issues? Did I do something wrong? Someone needs to be motivated to answer your questions.. Either because someone is paying them, or because they get some benefit out of spending the time. What's great about open source is anyone has access to the code, and you can exchange something other than money to get help with software. But there still needs to be something the person answering your question gets in return for spending the time and effort on it. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] multiple NLI's/interfaces to listen on?
On Tue, Aug 26, 2008 at 10:15:28AM -0400, murray smigel wrote: I have exactly the same problem. [EMAIL PROTECTED] ~]$ cat /etc/modprobe.conf alias scsi_hostadapter aacraid alias scsi_hostadapter1 sata_nv alias ib0 ib_ipoib alias ib1 ib_ipoib alias net-pf-27 ib_sdp options lnet networks=tcp0(eth0),tcp1(eth1),o2ib0(ib0) alias eth0 forcedeth alias eth1 forcedeth [EMAIL PROTECTED] ~]$ uname -a Linux nasmds 2.6.18-53.1.14.el5_lustre.1.6.5.1smp #1 SMP Wed Jun 18 19:45:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux I can mount via eth0, and ib0, but mounts via eth1 fail. If a client machine has both eth0 and eth1 networks enabled, then the mount via eth1 seems work, but if I ifdown eth0, then the eth1 mount won't work. I'm having similiar trouble with eth and IB.. I can 'mount -t lustre [EMAIL PROTECTED]:/lustre /lustre' , and it works but packets go over the ethernet interface, and performance is around 100Mb/sec. (gig-E speeds) FYI, I'm also running 2.6.18, from Debian. What OFED stack are you running BTW? ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] HAMMER
On Sat, Aug 23, 2008 at 05:51:36PM +0400, Nikita Danilov wrote: Mag Gam writes: Looks like there is another parallel filesystem similar to Lustre called HAMMER. http://kerneltrap.org/DragonFlyBSD/HAMMER_Filesystem_Design Hello, Has anyone heard about this? The architecture seems very similar to Lustre. while HAMMER design is very interesting and it looks M. Dillon plans to ultimately use it as a part of his single-image Dragonfly clustering, it's a local file system, and as such cannot be fairly compared to Lustre. I start feeling like an old geezer when I hear about a new filesystem of the day. Let's compare this once there's at least 1 top500 machine running hammer. What I really want to know is what happens to a hammer filesystem when a node starts randomly corrupting memory. Instant multi-master corruption replication!! The issues Hammer seems to try to solve in clustering seem to be a lot of the same issues I decided to try running AFS as the root filesystem. But It's not a parallel network filesystem. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre and ext4
On Fri, Aug 22, 2008 at 03:49:44PM +0400, Nikita Danilov wrote: Mag Gam writes: Hello, It seems Lustre uses ext4 on its backends (OST/MDS) and I was curious if kernel patching will be required in the future releases? I would be default Lustre configuration uses `ldiskfs' as a back-end file system, which is a modified ext4. ldiskfs is a separate file system type and a separate module, so no kernel patching is needed for it. Other parts of Lustre server do require changes to the core kernel code though. I would really like to see some wiki or other evolving technical documentation on what changes are really requried to the core kernel code, and why, and what barriers there are to getting those patches accepted into the mainline kernel. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] HLRN lustre breakdown
This is a big nasty issue, particularly for HPC applications where performance is a big issue. How does one even begin to benchmark the performance overhead of a parallel filesystem with checksumming? I am having nightmares over the ways vendors will try to play games with performance numbers. My suspicion is that whenever a parallel filesystem with checksumming is available and works, that all the end-users will just turn it off anyway because the applications will run twice as fast without it, regardless of what the benchmarks say.. leaving us back at the same problem. On Wed, Aug 20, 2008 at 07:12:10PM +0200, Bernd Schubert wrote: Oh damn, I'm always afraid of silent data corruptions due to bad harddisks. We also already had this issue, fortunately we found this disk before taking the system into production. Will lustre-2.0 use the ZFS checksum feature? Thanks, Bernd On Wednesday 20 August 2008 19:08:34 Peter Jones wrote: Hi there I got the following background information from Juergen Kreuels at SGI It turned out that a bad disk ( which did NOT report itself as being bad ) killed the lustre leading to data corruption due to inode areas on that disk. It was finally decided to remake the whole FS and only during that action we finally ( after nearly 48 h ) found that bad drive. It had nothing to do with the lustre FS itself. Lustre had been the victim of a HW failure on a Raid6 lun. I hope that this helps PJones Heiko Schroeter wrote: Hello list, does anyone has more background infos of what happened there ? Regards Heiko HLRN News - Since Mon Aug 18, 2008 12:00 HLRN-II complex Berlin is open for users, again. During the maintenance it turned out that the Lustre file system holding the users $WORK and $TMPDIR was damaged completely. The file system had to be reconstructed from scratch. All user data in $WORK are lost. We hope that this event remains an exception. SGI apologizes for this event. /Bka This is an announcement for all HLRN Users ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- Bernd Schubert Q-Leap Networks GmbH ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- -- Troy Benjegerdes'da hozer'[EMAIL PROTECTED] Somone asked me why I work on this free (http://www.gnu.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life. -- Charles Shultz ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] LBUG on client: Found existing inode ...?in?lock
I agree that hiding bugs is quite bad. rant I'm going to be an open source curmudgeon for a minute and say that if Sun/CFS wants to track customer-specific, sensitive data bugs, they need to have a separate system and pay someone to make sure that all internal bugs are santized and put into the open source project bug tracker. Sun/CFS gets a huge mindshare and market acceptance benefit from the open source project. Hiding bugs WILL kill that mindshare and acceptance benefit. If Lustre isn't a full first class public open source project, all the new and really innovative work will get done on competing open source filesystems. /rant Now back to real work ;) On Thu, Aug 21, 2008 at 04:40:54PM +0200, Erich Focht wrote: Thanks, Brian. A more general comment: what is the use of invisible bugs, anyway? I suppose the bug has been set private by the reporter. Wouldn't it actually make sense to have all bugs open, such that others are warned of the issue? Guess if somebody doesn't want to disclose the company on behalf of which the bug was reported, a mechanism for anonymizing the reporter would make more sense. Anyway, I feel like hiding bugs is bad in an open source project. Regards, Erich On Mittwoch 20 August 2008, Brian J. Murrell wrote: On Wed, 2008-08-20 at 16:47 +0200, Erich Focht wrote: Thanks but... I don't seem to be authorized to see that bug (?). Oh, yes. :-( I tend to forget to look at the privacy settings on bugs. Is that bug fixed in 1.6.5.1? No. Reported on 1.6.5 in fact. Any advice resulting from its content? None yet. It's been escalated to engineering though. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- -- Troy Benjegerdes'da hozer'[EMAIL PROTECTED] Somone asked me why I work on this free (http://www.gnu.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life. -- Charles Shultz ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] HLRN lustre breakdown
Actually, Lustre 1.6.5 does checksumming by default, and that is how we do our benchmarking. Some customers will turn it off because the overhead hurts them. New customers may not even notice it... Also, for many workloads the data integrity is much more important than the speed. I went digging in CVS HEAD for 'checksum', and it wasn't clear to me if this was end-to-end (from file write all the way to disk), or just an option for network RPC's. Is there some design or architecture document on the checksumming? All I could find was some references to the kerberos5 RPC checksums. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] LBUG on client: Found existing inode ...?in?lock
Since being part of Sun the Lustre designs and design discussions are available to the public (e.g. lustre-devel, wiki, public bugs for feature development, and internal debugging discussions, etc). In truth, it hasn't made a huge difference in external contributions to code or design, because the barrier is more to do with complexity than with being able to just read the document. manual.lustre.org and the other wikis are very nice. Keep up the good work. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] forcing o2ib vs TCP?
How do I force traffic over o2ib instead of TCP? Is there something I need to modify on the already existing filesystem? I thought just specifying the o2ib server properly would do it. (The filesystem was created with the MGS just listing the TCP address) ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre + debian
them up. You are now ready to use Lustre Debian style. If deploying this on a cluster, just install your ready made .debs for the kernel and modules. None of this was explained in the README in /usr/share/doc/Lustre so we had to figure it out ourselves. Hopefully the Debian folks have fixed that. Could you please have a look on our Readme.Debian. I've attached the current version of this README. My install (from lenny packages) is conspicuously missing both README.Debian and the 'lmc' tool da6:~# dpkg -l | grep lustre ii liblustre 1.6.4.3-1 ii linux-headers-2.6.18-lustre-1.6.5.1 20080818 ii linux-image-2.6.18-lustre-1.6.5.1 20080818 ii linux-patch-lustre1.6.4.3-1 ii lustre-source 1.6.4.3-1 ii lustre-tests 1.6.4.3-1 ii lustre-utils 1.6.4.3-1 ii openafs-modules-2.6.18-lustre-1.6.5.1 1.4.7.dfsg1-3~bpo40+1+20080818 What am I missing? ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre + debian
I was wondering what kernel works.. it looks like you are using 2.6.18, correct? What needs to happen to get 2.6.25 to work? I'd really like to have lustre and PVFS coexist on the same box. On Fri, Aug 15, 2008 at 02:33:40PM -0700, David Brown wrote: On Fri, Aug 15, 2008 at 9:48 AM, Robert LeBlanc [EMAIL PROTECTED] wrote: You should be able to use official Debian mirrors, that is all that we used, you don't have to build your own packages from scratch either as the packages are already in Lenny. Just apt-get the packages that he built: lustre-utils#Userspace lustre util lustre-dev #Development headers lustre-source #Source for the kernel module lustre-tests#Test suite linux-patch-lustre #Patch for the linux kernel. And the kernel-source and just skip down to where he builds his kernel. Robert Just thought I'd mention the package repository that I maintain as well for debian/ubuntu lustre packages http://www.pdsi-scidac.org/repository/debian http://www.pdsi-scidac.org/repository/ubuntu I rebuild the latest packages from debian and build them for all distributions (except for ubuntu/intrepid and debian/experimental) of debian and ubuntu. If you don't want to build everything on your own using mine is what its there for. Thanks, - David Brown On 8/15/08 10:38 AM, Troy Benjegerdes [EMAIL PROTECTED] wrote: I'm about to try this, and figured it would be worth documenting on the wiki.. http://wiki.lustre.org/index.php?title=Debian_Install so far the only issue is debian.internal.sanger.ac.uk is not visible to us outsiders ;) -- Robert LeBlanc Life Sciences Computer Support Brigham Young University [EMAIL PROTECTED] (801)422-1882 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss -- -- Troy Benjegerdes'da hozer'[EMAIL PROTECTED] Somone asked me why I work on this free (http://www.gnu.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life. -- Charles Shultz ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre + debian
On Mon, Aug 18, 2008 at 09:21:16AM -0700, David Brown wrote: Lustre developers only support vanilla 2.6.22.x ish (iirc). I don't think they've tried pushing forward the support to newer vanilla kernels. I think its about time for them to push forward their support for newer kernels. I'm sure the lustre package maintainers for debian would appreciate the effort. They do have patch sets for 2.6.23 but I'm unsure how stable they really are. Does anyone know the last time the Lustre kernel patches had some review on the linux kernel mailing list? I know there are some politics and egos involved, but generally the resulting code reviews and flames result in better code. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] debian package llmount.sh missing/broken
Where's the right place to report that the debian lenny packages have a broken llmount.sh? It would be nice if debconf could be used to automagically configure lustre in the same way that the Debian openafs packages will do some of the afs cell setup work. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] VMWare or VirtualBox..
There seem to be a couple of assumptions here that would be worth examining further. My first thought is that running Lustre under something like Xen might be very usefull for virtualization, failover, and load balancing. If there is some potential resource deadlock that could occur, there ought to be some diagram or documentation of what deadlocks are possible. It would be a lot of work to conclusively figure out what the potential deadlocks actually are.. But it seems like a worthwhile excercise, and running Lustre in a VM would be a good way to do QA testing so that new deadlocks are not added by code changes. On Fri, Aug 15, 2008 at 03:17:39PM -0700, Klaus Steden wrote: Hi Robert, You're likely to find that performance will suffer when run under a VM. Lustre makes pretty extensive use of all the resources at its disposal, and having to compete with physical devices under a VM that runs as an ordinary user process is more than likely going to lead to resource deadlocks. I wouldn't suggest running Lustre under VMs. cheers, Klaus ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
