[Lustre-discuss] Lustre Administration Support Level I training will be offered on July 15-17, 2008 in Beijing, China
Lustre Administration Support Level I training will be offered on July 15-17, 2008 in Beijing, China. Please click here for course description: https://www.suntrainingcatalogue.com/eduserv/client/loadCourse.do?coId=zh_CN_ES-288coCourseCode=ES-288l=zh_CN Course Language: Chinese Mandarin* Price:* RMB 9,800 for 3 days of in-depth technical hands-on instruction *Registration for Partners:* Please register from Kris Lye [EMAIL PROTECTED] *Registrations for Customers:* Please register from Jian Wu [EMAIL PROTECTED] Questions regarding Lustre courses can be sent to [EMAIL PROTECTED] Thank you! ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] add space of MDS problem?
On Fri, 2008-06-13 at 04:35 -0700, Johnlya wrote: I tested it and it can work. But I don't do it by its method. I want to add space like adding OST. That is currently not possible. The only method we support for MDT expansion is the backup/recreate MDT/restore process. Cheers, b. signature.asc Description: This is a digitally signed message part ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Rule of thumb for setting up lustre resources...
On Sat, 2008-06-14 at 14:22 -0400, Mark True wrote: Hello! Hi. A If increasing the number of OSTs increases throughput, is there a relationship that can be used to determine how many OSTs we're likely to need at the outset to establish a baseline minimum throughput. Of course. For examples, if I want to get 3GB sustained throughput how many OSTs will facilitate this. That is _completely_ dependent on your hardware configuration. If you are adding an identical (to an existing) OSTs you can simply use the speed of the existing OST to determine how much more the new OST will add. But be very careful of ceilings. You can of course only add so many OSTs before you start to hit other resource limitations such as bus bandwidth in the OSS and network bandwidth of the OSS's interconnect, etc. In short, you need to understand the performance capability of all of your components to come up with an overall design that meets your performance goals and scales to future goals. B Does the MGS and MDS have to be separate for best performance, or can they be consolidated into one server without causing too much hardship I'd tend to say that most people put them into the same server. For anything but toy installations however, we strongly suggest you put the MGS and MDT on separate devices. C Right now I am looking at a model where I am connecting all the OSTs, and the MDS/MGS together using infiniband, Just to keep the nomenclature straight, an OST is a device (i.e. a disk) in/attached to an OSS. An OSS is the server that serves OSTs. and connecting the storage via fibrechannel. Is this the ideal solution or am I going in the wrong direction. That sounds suitable. b. signature.asc Description: This is a digitally signed message part ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Announce: Lustre 1.6.5 is available!
On Mon, 2008-06-16 at 00:25 +0200, Erich Focht wrote: A related question: is there an easy way of building the Lustre kernel RPMs for the RHEL 5.2 kernel? Which kernel version is RHEL 5.2? b. signature.asc Description: This is a digitally signed message part ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Announce: Lustre 1.6.5 is available!
Current version of rhel 5.2 is: [EMAIL PROTECTED] ~]# rpm -qa | grep kernel kernel-xen-2.6.18-92.1.1.el5 I'm running xen stuff currently... On Mon, Jun 16, 2008 at 7:48 AM, Brian J. Murrell [EMAIL PROTECTED] wrote: On Mon, 2008-06-16 at 00:25 +0200, Erich Focht wrote: A related question: is there an easy way of building the Lustre kernel RPMs for the RHEL 5.2 kernel? Which kernel version is RHEL 5.2? b. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Gluster then DRBD now Lustre?
I have been spending a lot of time with Gluster, I like it a lot, on the surface it looks great. I like that I can get RAID 6 like functionality out of it, however after testing found it is not just ready for prime time. Our day one config is two servers with 10TB each in NYC and SJC. Originally the plan was to active/active mirror them, but even with gig e the delay kills your write speed. Since Gluster did not work out we started testing DRBD. The plan was to active/active mirror the two servers in each site and then setup scripts to copy the data we need between sites. When we need more servers, we would add them in groups of two and use Gluster (hoping it is ready in 4 - 6 months) to unify the DRBD groups into a larger shared namespace. This is working in a test setup, however there are some down sides. The first is that DRBD only supports IP, so we have to run IPoIB over our our infiniband adapters, not an ideal solution. The second is that we are using infiniband adapters on centos 5.1 xen kernel and can't bind them together because we need OFED 3 and it removed bind because it now is in the kernel, but not in 2.6.18 we need for xen. Anyway, my question is should I run Luster instead of DRBD and is there any time frame for RAID 6 like functionally out of Lustre? P.S. Once long long ago and far far away lustre had links that you could download software from. Today the only way I see to do it is to log into sun and then download. I have scripts that I use to build stuff and this is a big pain Nathan StrattonCTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.nethttp://www.blinkmind.com ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Rule of thumb for setting up lustre resources...
Hi Mark, See my comments inline below. cheers, Klaus On 6/14/08 11:22 AM, Mark True [EMAIL PROTECTED]did etch on stone tablets: Hello! I am new to the list, but I have been researching Lustre for quite some time and finally have an occasion to use it. I am trying to do some capacity planning and I am wondering if there are some general rules of thumb for configuring a Lustre environment. Specifically: A If increasing the number of OSTs increases throughput, is there a relationship that can be used to determine how many OSTs we're likely to need at the outset to establish a baseline minimum throughput. For examples, if I want to get 3GB sustained throughput how many OSTs will facilitate this. B Does the MGS and MDS have to be separate for best performance, or can they be consolidated into one server without causing too much hardship C Right now I am looking at a model where I am connecting all the OSTs, and the MDS/MGS together using infiniband, and connecting the storage via fibrechannel. Is this the ideal solution or am I going in the wrong direction. This is a good solution, and will give you good performance overall, although you can mix different storage technologies and network technologies within the same storage environment and it should remain relatively transparent. I've got a cluster that handles both FC storage and iSCSI storage, but I know there are people out there using DRBD, and I'm dying to try Infiniband-based storage as well. Anything that presents a block device to an OSS should be suitable for use with Lustre, but some will perform better than others. Bottom line, I think, is pick the best technology for your price range and performance needs. Infiniband + FC is pretty much the top of the mountain, though. D Just wondering what clustering software people use on the front end with Lustre typically, if they are going to be using this as a filesystem for some kind of HPC environment, what is the most popular clustering technology for this. Our CFS clusters are all organized as part of ROCKS clusters. I know a number of people on this list are on the ROCKS list, so there's good cross-pollination between technologies. It's a mature cluster architecture designed for HPC, and bundles a number of useful solutions and tools onboard (MPI, SGE, Torque, distributed compilers, visualization, etc.). It's also relatively easy to integrate with Lustre, as you can simply drop in the pre-built Lustre RPMs into the cluster installer and be ready to go in a few minutes. E Does Heartbeat install next to whatever HPC clustering technology you have? I'm using Linux-HA, and it wasn't built into my cluster software distro, but it was easy enough to drop into the mix, and as of late last year had native disk support for Lustre file systems. Thanks, and I hope that I can soon be someone who contributes rather than just asking questions :) --Mark T. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Gluster then DRBD now Lustre?
[EMAIL PROTECTED] wrote: On Mon, 16 Jun 2008, Kilian CAVALOTTI wrote: On Monday 16 June 2008 11:40:40 am Andreas Dilger wrote: NYC == New York City? What is SJC? SJC == San Jose, California That's why I thought, but if so, the following part loses me: This is working in a test setup, however there are some down sides. The first is that DRBD only supports IP, so we have to run IPoIB over our our infiniband adapters, not an ideal solution. Nathan, you won't be able to use Infiniband between Ney Work City and San Jose, CA, anyway, right? Even without considering IB cables' length limitation, and unless you can use some kind of dedicated, special-purpose link between your sites, the public Internet is not really able to provide bandwidth nor latencies compatible with Infiniband standards. Ok, so in the original email east to west was what we originally wanted to do but realized that would not be possible because of round trip delay even over gig e. Instead of mirroring our traffic east west we are starting with 2 servers in each location tied together with Infiniband. The infiniband cables are only 5M. : ) Currently we are mirroring traffic with DRBD between the two local systems in each datacenter, but we are looking for the tradeoffs of switching to Lustre since DRBD does not support Infiniband. UmmLustre is not a replacement for DRBD, so we're very confused over here. Lustre is a way of making a big distributed filesystem out of a bunch of storage nodes. We don't do replication, it's basically RAID 0. So, you could use Lustre to make one big filesystem out of two local servers. You could even make one big filesystem out of your multiple locations over the WAN (it's been done). But, you can't use Lustre to mirror data. (yet, wait a year) So I think your Gluster expedition might have confused you. Gluster and Lustre are only words that sound somewhat the same, there is _no_ relationship between the two. (except the fact that there is some filesystem goop involved) You're comparing apples to knee socks if you are attempting to map gluster experience to a Lustre setup. cliffw -Nathan ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] 1.6.5 and OFED?
Is there any word on when the IB packages might be making it up to the download site for 1.6.5? As had been previously noted, they were missing when the rest of 1.6.5 was pushed. Thanks, --Joe ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] How do I recover files from partial lustre disk?
Greetings! I am using Lustre 2.6.18-53.1.13.el5_lustre.1.6.4.3smp kernel on a CentOS 5 linux x86_64 linux box. We had a hardware problem that caused the underlying ext3 partition table to completely blow up. This is resulting in only three of five OSTs being mountable. The main lustre disk of this unit cannot be mounted because the MDS knows that two of its parts are missing. The underlying set-up is JBOD hw that is passed to the linux OS, via an LSI ELP card in this case, as a simple device, ie. sde, sdf,...The simple devices were partitioned using parted and formatted ext3 then lustre was built on top of the five ext3 units. There was no striping done across units/JBODS. Three of the five units passed an e2fsck and an lfsck. Those remaining units are mounted as such: /dev/sdc 13T 6.3T 5.7T 53% /srv/lustre/OST/crew4- OST0003 /dev/sdd 13T 6.3T 5.7T 53% /srv/lustre/OST/crew4- OST0004 /dev/sdf 13T 6.2T 5.8T 52% /srv/lustre/OST/crew4- OST0001 Being that it is unlikely that we shall be able to recover the underlying ext3 on the other two units, is there some method by which I might try to rescue the data from these last three units mounted currently on the OSS? Any and all suggestion genuinely appreciated. megan ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
