Re: [Lustre-discuss] Rebuilding kernel rpms
On Thu, 2010-12-16 at 13:17 +0100, Arne Brutschy wrote: > Hello, Hi, > I need PAE-enabled kernels for my cluster. Do you actually use a 32 bit kernel/OS installation on your Lustre _servers_? Can I ask why (not use an x86_64 kernel)? b. signature.asc Description: This is a digitally signed message part ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] LUG 2011 - Registration is Now Open
LUG 2011 will be held in Orlando, Florida from Tuesday, April 12, 2011 through April 14, 2011 at the Marriott World Center Golf and Spa resort. This two-and-a-half-day event is the primary venue for discussion and seminars on open source parallel file system technologies with a unique focus on the Lustre parallel file system. The conference is generously supported by the following corporate sponsors: Bull, DataDirect Networks, Dell, HP, LSI, Oracle, SGI, and Whamcloud. REGISTRATION IS NOW OPEN Members of the Lustre community can now register for LUG 2011 via the conference website at http://www.olcf.ornl.gov/event/lug-2011/. Early bird registration (through March 15) is $400 per person, while standard registration (after March 15) is $550 per person for the entire two-and-a-half-day event. We encourage all those who plan to attend to register as early as possible allowing us to effectively plan and budget for this event. CALL FOR PARTICIPATION The LUG program committee would like to invite members of the Lustre community to submit presentation abstracts for inclusion in this year’s meeting. It is not necessary to submit a technical paper, just an abstract of the proposed talk no more than a page in length. Talks should be approximately half an hour in length and reflect the work that you or your site are doing with Lustre. The deadline to submit presentation abstracts is February 14, 2011. For questions or to submit a presentation abstract contact the program committee chair, Stephen Simms, at lug-submissi...@ornl.gov. LUG 2011 is user-led with an organizing committee made up of representatives from Commissariat à l''énergie atomique (CEA), Indiana University – Pervasive Technology Institute (IU), Lawrence Livermore National Laboratory (LLNL), Naval Research Laboratory (NRL), ORNL, Sandia National Laboratories (SNL), and Texas Advanced Computing Center (TACC). http://www.olcf.ornl.gov/event/lug-2011/ ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre community build server
Hi, On Dec 16, 2010, at 9:42 , DEGREMONT Aurelien wrote: > Hi > > Robert Read a écrit : >> Hi Aurélien, >> >> Yes, we've noticed Hudson's support for testing is not quite what we need, >> so we're planning to use Hudson to trigger our testing system, but not >> necessarily to manage it. We'd definitely be interested in learning more >> about your experiences, though. > I do not know what you mean by triggering your testing system. But here is > what I set up. I mean that once the build is complete we will notify the test system that new build is ready to be picked up and tested. We haven't yet implemented that part of it. > Hudson has only 1 slave node dedicated to testing Lustre 2. > Hudson will launch a shell script through ssh to it. > > This script: > - retrieves Lustre source (managed by Hudson git plugin) > - compiles it. > - launches acceptance-small with several parameters. > - acceptance-small will connect to other nodes dedicated for these tests. > > acc-sm have been patched: > - to be more error resilient (does not stop at first failure) > - to generate a test report in JUNIT format. > > Hudson fetch the junit report and parse it thanks to its plugin. > Hudson can display in its interface all tests successes and failures. > > Everything goes fine as long as: > - the testsuite leaves the node in a good shape. It is difficult to have a > automatic way to put the node back. Currently, we need to manualy fix that. > - Hudson does not know about the other nodes used by acc-sm. And so can > trigger tests even if some sattelites nodes are unavailable. > > How is you do this on your side? We don't plan to use Hudson to manage our testing results as I don't think it would scale very well for all the testing we might do for each build. We're currently building a more custom results server that's similar (in spirit at least) to the kinds of tools we had at Oracle. We'll make it available once it's in presentable form. Actually, our first step was to replace the acceptance-small.sh driver script with one that has a more sensible user interface for running the standard tests. Since the test-framework.sh on master has already been changed to produce test results in yaml format, the new script collects these with the logs, and is capable of submitting them to the test results server. Currently this is being run manually, though. Automating the test execution and connecting all the pieces will be next step. cheers, robert > > > Aurélien > >> robert >> >> >> >> >> On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote: >> >> >>> Hi Robert, >>> >>> That's very interesting. >>> At CEA we also have a Hudson platform and I'm running acceptance-small for >>> several Lustre branches in it. Hudson is a great tool but it was not design >>> to test tools that run kernel-space that can crash your nodes or, at least, >>> put your kernel in a bad shape. I will be very interested to share Hudson >>> experience testing Lustre and see how you've configured it for your own >>> tests. >>> >>> >>> Aurélien >>> >>> Robert Read a écrit : >>> Hi, As I mentioned the other day, Whamcloud is hosting a Hudson build server and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for both 1.8.x and 2.x branches. Our intention is for this to be a resource for the Lustre community to find recent Lustre packages for variety of Linux distributions. Early next year we'll connect this to our test system so at least some of the packages can be tested, as well. We would be interested in hearing from anyone that would like to participate producing builds. Hudson is an distributed system, and it's easy to add more build nodes, even behind firewalls (some of us are running build VMs on our home machines). If you would like add another distribution or architecture we don't have yet, or even one we do have (the more the merrier), we'd be happy to work with you to do that. Please contact me if you are interested. cheers, robert ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> > ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre community build server
Hi Robert Read a écrit : > Hi Aurélien, > > Yes, we've noticed Hudson's support for testing is not quite what we need, so > we're planning to use Hudson to trigger our testing system, but not > necessarily to manage it. We'd definitely be interested in learning more > about your experiences, though. > I do not know what you mean by triggering your testing system. But here is what I set up. Hudson has only 1 slave node dedicated to testing Lustre 2. Hudson will launch a shell script through ssh to it. This script: - retrieves Lustre source (managed by Hudson git plugin) - compiles it. - launches acceptance-small with several parameters. - acceptance-small will connect to other nodes dedicated for these tests. acc-sm have been patched: - to be more error resilient (does not stop at first failure) - to generate a test report in JUNIT format. Hudson fetch the junit report and parse it thanks to its plugin. Hudson can display in its interface all tests successes and failures. Everything goes fine as long as: - the testsuite leaves the node in a good shape. It is difficult to have a automatic way to put the node back. Currently, we need to manualy fix that. - Hudson does not know about the other nodes used by acc-sm. And so can trigger tests even if some sattelites nodes are unavailable. How is you do this on your side? Aurélien > robert > > > > > On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote: > > >> Hi Robert, >> >> That's very interesting. >> At CEA we also have a Hudson platform and I'm running acceptance-small for >> several Lustre branches in it. Hudson is a great tool but it was not design >> to test tools that run kernel-space that can crash your nodes or, at least, >> put your kernel in a bad shape. I will be very interested to share Hudson >> experience testing Lustre and see how you've configured it for your own >> tests. >> >> >> Aurélien >> >> Robert Read a écrit : >> >>> Hi, >>> >>> As I mentioned the other day, Whamcloud is hosting a Hudson build server >>> and producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it >>> works) for both 1.8.x and 2.x branches. Our intention is for this to be a >>> resource for the Lustre community to find recent Lustre packages for >>> variety of Linux distributions. Early next year we'll connect this to our >>> test system so at least some of the packages can be tested, as well. >>> >>> We would be interested in hearing from anyone that would like to >>> participate producing builds. Hudson is an distributed system, and it's >>> easy to add more build nodes, even behind firewalls (some of us are running >>> build VMs on our home machines). If you would like add another distribution >>> or architecture we don't have yet, or even one we do have (the more the >>> merrier), we'd be happy to work with you to do that. Please contact me if >>> you are interested. >>> cheers, >>> robert >>> >>> >>> >>> >>> ___ >>> Lustre-discuss mailing list >>> Lustre-discuss@lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> > > ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre community build server
Hi Aurélien, Yes, we've noticed Hudson's support for testing is not quite what we need, so we're planning to use Hudson to trigger our testing system, but not necessarily to manage it. We'd definitely be interested in learning more about your experiences, though. robert On Dec 16, 2010, at 1:22 , DEGREMONT Aurelien wrote: > Hi Robert, > > That's very interesting. > At CEA we also have a Hudson platform and I'm running acceptance-small for > several Lustre branches in it. Hudson is a great tool but it was not design > to test tools that run kernel-space that can crash your nodes or, at least, > put your kernel in a bad shape. I will be very interested to share Hudson > experience testing Lustre and see how you've configured it for your own tests. > > > Aurélien > > Robert Read a écrit : >> Hi, >> >> As I mentioned the other day, Whamcloud is hosting a Hudson build server and >> producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for >> both 1.8.x and 2.x branches. Our intention is for this to be a resource for >> the Lustre community to find recent Lustre packages for variety of Linux >> distributions. Early next year we'll connect this to our test system so at >> least some of the packages can be tested, as well. >> >> We would be interested in hearing from anyone that would like to participate >> producing builds. Hudson is an distributed system, and it's easy to add more >> build nodes, even behind firewalls (some of us are running build VMs on our >> home machines). If you would like add another distribution or architecture >> we don't have yet, or even one we do have (the more the merrier), we'd be >> happy to work with you to do that. Please contact me if you are interested. >> >> cheers, >> robert >> >> >> >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
Ok. I use package on http://www.pdsi-scidac.org/repository/debian/ and it's not 1.8 but 1.6 (see on luster-source package) I'll try with good sources (1.8) and reply after Thanks a lot -Message d'origine- De : Andreas Dilger [mailto:andreas.dil...@oracle.com] Envoyé : jeudi 16 décembre 2010 16:57 À : Eudes PHILIPPE Cc : 'Wang Yibin'; 'lustre-discuss' Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-16, at 08:25, Eudes PHILIPPE wrote: > If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload > is ok (sftp said nothing), but the file is different (the md5sum is > different!) It's very dangerous!! What version of Lustre are you using? This should definitely not happen (md5sum is different, without any errors). Is the size of the new file in Lustre matching the size of the original file? Does it work OK if you are using "cp" instead of sftp? > Is there a solution, if I see all ossX are almost full, when I add (some) new > oss, distribute data on these new oss to always have the same poucentage use > on all oss? This already works today, so long as the file sizes are not so large that they fill most of an OST. > -Message d'origine- > De : Wang Yibin [mailto:wang.yi...@oracle.com] Envoyé : jeudi 16 > décembre 2010 16:09 À : Eudes PHILIPPE Cc : lustre-discuss Objet : Re: > [Lustre-discuss] howto make a lvm, or virtual lvm? > > > 在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： > >> Ok, so, i'll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> * >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn't write to remote file "/home/big_log.log": Failure) >> ** > > All files in a directory inherits its parent dir's stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be > written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it's ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) > deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe > offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -Message d'origine- >> De : Andreas Dilger [mailto:andreas.dil...@oracle.com] >> Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : >> lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don't >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Unable to mount OSTs
I am focusing my effort on the simplest lustre filesystem consisting of a single OST. I can mount the MDT/MGS and the OST with no apparent errors. However, when a client is booting and trying to mount, I start getting a log entry on the MGS: Dec 16 08:10:45 nas-0-0 kernel: LustreError: 5732:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-11) r...@8103dfa61400 x1355301045403661/t0 o38->@:0/0 lens 368/0 e 0 to 0 dl 1292515945 ref 1 fl Interpret:/0/0 rc -11/0 The client loops giving LustreError: 11-0. The mds_connect operation failed with -11. Brian Andrus -Original Message- From: Wang Yibin [mailto:wang.yi...@oracle.com] Sent: Thursday, December 16, 2010 6:38 AM To: Andrus, Brian Contractor Cc: lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Unable to mount OSTs What does dmesg tell you after you mounted that particular OST? You can find some clue there on the OSS and (maybe) MGS. 在 2010-12-16，下午10:30， Andrus, Brian Contractor 写道： > Thank you for your assistance. > I am almost there. > I am able to get all the OSTs to mount, but when I do "lctl dl", one of them > is not showing up for the work filesystem. > > I can ping all OSSs from each other and once I have done that see them all in > a peer_list. I have even tried mounting the OST on an OSS that already has > another OST that is being recognized. > > Brian Andrus > > -Original Message- > From: Wang Yibin [mailto:wang.yi...@oracle.com] > Sent: Thursday, December 16, 2010 12:29 AM > To: Andrus, Brian Contractor > Cc: Andreas Dilger; lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > This error sounds like that there is some unrecognized parameter(s) on your > OSTs. > Please check your OST parameters. > You hit this issue mostly because we have some fix disallowing unrecognized > parameters (22386). > > 在 2010-12-16，下午3:56， Andrus, Brian Contractor 写道： > >> Well e2fsck didn't help. Same error for those two (Function not >> implemented). >> I also see in the MGS/MDS log each time I try: >> LustreError: 5974:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed >> to write nscratch-OST0001 log (-38) >> LustreError: 5973:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed >> to write nscratch-OST0002 log (-38) >> >> >> >> Brian Andrus >> >> -Original Message- >> From: lustre-discuss-boun...@lists.lustre.org >> [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrus, >> Brian Contractor >> Sent: Wednesday, December 15, 2010 11:48 PM >> To: Andreas Dilger >> Cc: lustre-discuss@lists.lustre.org >> Subject: Re: [Lustre-discuss] Unable to mount OSTs >> >> I actually did run writeconf on the MGS/MDS as well. >> I just did it again to test. >> >> Followed the instructions in section 4.3.11 in the luster manual. >> Mounting MDT goes fine. >> Mounting the OSTs, the first just seems to be hanging. The other two >> both gave: >> [r...@ost2 ~]# mount -t lustre /dev/VG_lustre/scratch_ost2 >> /mnt/lustre/scratch/ost2/ >> mount.lustre: mount /dev/VG_lustre/scratch_ost2 at >> /mnt/lustre/scratch/ost2 failed: Function not implemented >> >> Currently running e2fsck on the two OSTs that didn't want to mount. I >> have already run it, but doing so again, just in case. >> >> Brian Andrus >> >> -Original Message- >> From: Andreas Dilger [mailto:andreas.dil...@oracle.com] >> Sent: Wednesday, December 15, 2010 10:55 PM >> To: Andrus, Brian Contractor >> Cc: lustre-discuss@lists.lustre.org >> Subject: Re: [Lustre-discuss] Unable to mount OSTs >> >> On 2010-12-15, at 22:50, Andrus, Brian Contractor wrote: >>> I did change IPs of the NID for 2 or our lustre filesystems. >>> Made appropriate changes to each ost. One thing I did that I probably >> shouldn't have was doing a --writeconf on the OSTs. Now I see: >>> >>> Flags: 0x142 >>> (OST update writeconf ) >>> and I cannot mount them. I get: >>> >>> mount.lustre: mount /dev/VG_lustre/scratch_ost1 at >> /mnt/lustre/scratch/ost1 failed: Function not implemented >>> however I do seem to be able to mount the OSTs that I didn't do >> --writeconf on. >> >> You can binary edit this flag in the "CONFIGS/mountdata" file on each >> OST. >> Alternately, you could follow the process in the manual, and run >> writeconf on the MGS and MDS as well, and then the OSTs should be able >> to register again. >> >>> I see in syslog: >>> LustreError: 6182:0:(obd_mount.c:1097:server_start_targets()) Required >> registration failed for nscratch-OST0001: -38 >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailm
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
On 2010-12-16, at 08:25, Eudes PHILIPPE wrote: > If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok > (sftp said nothing), but the file is different (the md5sum is different!) > It's very dangerous!! What version of Lustre are you using? This should definitely not happen (md5sum is different, without any errors). Is the size of the new file in Lustre matching the size of the original file? Does it work OK if you are using "cp" instead of sftp? > Is there a solution, if I see all ossX are almost full, when I add (some) new > oss, distribute data on these new oss to always have the same poucentage use > on all oss? This already works today, so long as the file sizes are not so large that they fill most of an OST. > -Message d'origine- > De : Wang Yibin [mailto:wang.yi...@oracle.com] > Envoyé : jeudi 16 décembre 2010 16:09 > À : Eudes PHILIPPE > Cc : lustre-discuss > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > > 在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： > >> Ok, so, i'll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> * >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn't write to remote file "/home/big_log.log": Failure) >> ** > > All files in a directory inherits its parent dir's stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be > written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it's ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) > deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe > offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -Message d'origine- >> De : Andreas Dilger [mailto:andreas.dil...@oracle.com] >> Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : >> lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don't >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
Lustre has its own load-balancing algorithms (either round-robin or weighted) depending on the space usage of the OSTs. 在 2010-12-16，下午11:25， Eudes PHILIPPE 写道： > I was wrong about what I say > If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok > (sftp said nothing), but the file is different (the md5sum si different!) > It's very dangerous!! > > Is there a solution, if I see all ossX are almost full, when I add (some) new > oss, distribute data on these new oss to always have the same poucentage use > on all oss? > > > > -Message d'origine- > De : Wang Yibin [mailto:wang.yi...@oracle.com] > Envoyé : jeudi 16 décembre 2010 16:09 > À : Eudes PHILIPPE > Cc : lustre-discuss > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > > 在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： > >> Ok, so, i'll try this : >> - One mds >> - 2 physical oss with one drive (1 GB) (one ost on one oss) >> >> On client, mount mds on /home.. >> lfs setstripe -c2 /home >> >> I upload (in sftp) one file, 300 MB >> - On Oss 1, he use 150 Mb of 1000 >> - On Oss 2, he use 150 Mb of 1000 >> >> All right! >> >> I continue... copy my first file 4 times (so there is 5 * 300 MB = >> 1500 MB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> >> * >> Now, I add a new oss server, with one ost (1GB) >> - On Oss 1, he use 750 Mb of 1000 >> - On Oss 2, he use 750 Mb of 1000 >> - On Oss 3, he use 0 Mb of 1000 >> >> lfs setstripe -c3 /home on client >> >> I upload a big file, 1.3 Go >> He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop >> (Couldn't write to remote file "/home/big_log.log": Failure) >> ** > > All files in a directory inherits its parent dir's stripe attributes. > As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be > written to 3 objects located in different OSTs. > As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. > >> >> So now, >> - On Oss 1, he use 1000 Mb of 1000 >> - On Oss 2, he use 1000 Mb of 1000 >> - On Oss 3, he use 250 Mb of 1000 >> I upload again, just for see, my first file (300 MB), he copy the file >> only on Oss3 (oss 1 and 2 are full of course), it's ok :) >> >> Is there a solution for this problem? > > If you want to do write with system that has full OSTs, you need to either 1) > deactivate the full OSTs, or 2) set stripe size and offset properly. > In your specific case, get the stripe size of your file to 1 and stripe > offset to 2 (assuming the non-full OST index is 2). > >> >> Regards >> >> >> >> -Message d'origine- >> De : Andreas Dilger [mailto:andreas.dil...@oracle.com] >> Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : >> lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto >> make a lvm, or virtual lvm? >> >> On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >>> At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 >> physical raid 5 on each ostX and a big lvm I can extend as I want >> >> Lustre itself can not do RAID over the network, if that is what you >> are looking for... >> >>> For my first test, I upload on client a file (1.8 Go) (each ost have >>> 1 >>> Go) The problem, is, when sdb is full, he stop the copy, and don't >>> continue on ost2 >> >> If you create your file to be striped over both OSTs, then it should work. >> >> Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
I was wrong about what I say If oss1 and 2 are full, oss3 is ok, and if I sent a new file, upload is ok (sftp said nothing), but the file is different (the md5sum si different!) It's very dangerous!! Is there a solution, if I see all ossX are almost full, when I add (some) new oss, distribute data on these new oss to always have the same poucentage use on all oss? -Message d'origine- De : Wang Yibin [mailto:wang.yi...@oracle.com] Envoyé : jeudi 16 décembre 2010 16:09 À : Eudes PHILIPPE Cc : lustre-discuss Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? 在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： > Ok, so, i'll try this : > - One mds > - 2 physical oss with one drive (1 GB) (one ost on one oss) > > On client, mount mds on /home.. > lfs setstripe -c2 /home > > I upload (in sftp) one file, 300 MB > - On Oss 1, he use 150 Mb of 1000 > - On Oss 2, he use 150 Mb of 1000 > > All right! > > I continue... copy my first file 4 times (so there is 5 * 300 MB = > 1500 MB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > > * > Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn't write to remote file "/home/big_log.log": Failure) > ** All files in a directory inherits its parent dir's stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. > > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file > only on Oss3 (oss 1 and 2 are full of course), it's ok :) > > Is there a solution for this problem? If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). > > Regards > > > > -Message d'origine- > De : Andreas Dilger [mailto:andreas.dil...@oracle.com] > Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : > lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto > make a lvm, or virtual lvm? > > On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >> At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 > physical raid 5 on each ostX and a big lvm I can extend as I want > > Lustre itself can not do RAID over the network, if that is what you > are looking for... > >> For my first test, I upload on client a file (1.8 Go) (each ost have >> 1 >> Go) The problem, is, when sdb is full, he stop the copy, and don't >> continue on ost2 > > If you create your file to be striped over both OSTs, then it should work. > > Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
在 2010-12-16，下午10:49， Eudes PHILIPPE 写道： > Ok, so, i'll try this : > - One mds > - 2 physical oss with one drive (1 GB) (one ost on one oss) > > On client, mount mds on /home.. > lfs setstripe -c2 /home > > I upload (in sftp) one file, 300 MB > - On Oss 1, he use 150 Mb of 1000 > - On Oss 2, he use 150 Mb of 1000 > > All right! > > I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > > * > Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn't write to remote file "/home/big_log.log": Failure) > ** All files in a directory inherits its parent dir's stripe attributes. As you set the mountpoint dir to stripe over 3 OSTs, all files in it will be written to 3 objects located in different OSTs. As OST 1 and 2 are full, surely you'll get write failure with ENOSPC. > > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file only > on Oss3 (oss 1 and 2 are full of course), it's ok :) > > Is there a solution for this problem? If you want to do write with system that has full OSTs, you need to either 1) deactivate the full OSTs, or 2) set stripe size and offset properly. In your specific case, get the stripe size of your file to 1 and stripe offset to 2 (assuming the non-full OST index is 2). > > Regards > > > > -Message d'origine- > De : Andreas Dilger [mailto:andreas.dil...@oracle.com] > Envoyé : mercredi 15 décembre 2010 22:39 > À : Eudes PHILIPPE > Cc : lustre-discuss@lists.lustre.org > Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? > > On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: >> At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 > physical raid 5 on each ostX and a big lvm I can extend as I want > > Lustre itself can not do RAID over the network, if that is what you are > looking for... > >> For my first test, I upload on client a file (1.8 Go) (each ost have 1 >> Go) The problem, is, when sdb is full, he stop the copy, and don't >> continue on ost2 > > If you create your file to be striped over both OSTs, then it should work. > > Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
On 2010-12-16, at 7:49, Eudes PHILIPPE wrote: > Now, I add a new oss server, with one ost (1GB) > - On Oss 1, he use 750 Mb of 1000 > - On Oss 2, he use 750 Mb of 1000 > - On Oss 3, he use 0 Mb of 1000 > > lfs setstripe -c3 /home on client > > I upload a big file, 1.3 Go > He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop > (Couldn't write to remote file "/home/big_log.log": Failure) > ** > > So now, > - On Oss 1, he use 1000 Mb of 1000 > - On Oss 2, he use 1000 Mb of 1000 > - On Oss 3, he use 250 Mb of 1000 > I upload again, just for see, my first file (300 MB), he copy the file only > on Oss3 (oss 1 and 2 are full of course), it's ok :) > > Is there a solution for this problem? This is much less of a problem in real usage of Lustre, when the size of the file is not a large fraction of thectotal OST size. Lustre will balance object allocation dynamically based on how full an OST is, but cannot currently do this in the middle of a single file. With typical OST sizes in the 8-16TB range today, this is rarely a problem. Cheers, Andreas ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] howto make a lvm, or virtual lvm?
Ok, so, i'll try this : - One mds - 2 physical oss with one drive (1 GB) (one ost on one oss) On client, mount mds on /home.. lfs setstripe -c2 /home I upload (in sftp) one file, 300 MB - On Oss 1, he use 150 Mb of 1000 - On Oss 2, he use 150 Mb of 1000 All right! I continue... copy my first file 4 times (so there is 5 * 300 MB = 1500 MB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 * Now, I add a new oss server, with one ost (1GB) - On Oss 1, he use 750 Mb of 1000 - On Oss 2, he use 750 Mb of 1000 - On Oss 3, he use 0 Mb of 1000 lfs setstripe -c3 /home on client I upload a big file, 1.3 Go He write on oss1, 2 and 3, but, when oss 1 and oss2 are full, he stop (Couldn't write to remote file "/home/big_log.log": Failure) ** So now, - On Oss 1, he use 1000 Mb of 1000 - On Oss 2, he use 1000 Mb of 1000 - On Oss 3, he use 250 Mb of 1000 I upload again, just for see, my first file (300 MB), he copy the file only on Oss3 (oss 1 and 2 are full of course), it's ok :) Is there a solution for this problem? Regards -Message d'origine- De : Andreas Dilger [mailto:andreas.dil...@oracle.com] Envoyé : mercredi 15 décembre 2010 22:39 À : Eudes PHILIPPE Cc : lustre-discuss@lists.lustre.org Objet : Re: [Lustre-discuss] howto make a lvm, or virtual lvm? On 2010-12-15, at 10:06, Eudes PHILIPPE wrote: > At the end, I want (if it's possible), a raid 5 over Ethernet, or, 1 physical raid 5 on each ostX and a big lvm I can extend as I want Lustre itself can not do RAID over the network, if that is what you are looking for... > For my first test, I upload on client a file (1.8 Go) (each ost have 1 > Go) The problem, is, when sdb is full, he stop the copy, and don't > continue on ost2 If you create your file to be striped over both OSTs, then it should work. Use "lfs setstripe -c2 /home/newfile" to specify a stripe count of 2. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Unable to mount OSTs
What does dmesg tell you after you mounted that particular OST? You can find some clue there on the OSS and (maybe) MGS. 在 2010-12-16，下午10:30， Andrus, Brian Contractor 写道： > Thank you for your assistance. > I am almost there. > I am able to get all the OSTs to mount, but when I do "lctl dl", one of them > is not showing up for the work filesystem. > > I can ping all OSSs from each other and once I have done that see them all in > a peer_list. I have even tried mounting the OST on an OSS that already has > another OST that is being recognized. > > Brian Andrus > > -Original Message- > From: Wang Yibin [mailto:wang.yi...@oracle.com] > Sent: Thursday, December 16, 2010 12:29 AM > To: Andrus, Brian Contractor > Cc: Andreas Dilger; lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > This error sounds like that there is some unrecognized parameter(s) on your > OSTs. > Please check your OST parameters. > You hit this issue mostly because we have some fix disallowing unrecognized > parameters (22386). > > 在 2010-12-16，下午3:56， Andrus, Brian Contractor 写道： > >> Well e2fsck didn't help. Same error for those two (Function not >> implemented). >> I also see in the MGS/MDS log each time I try: >> LustreError: 5974:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed >> to write nscratch-OST0001 log (-38) >> LustreError: 5973:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed >> to write nscratch-OST0002 log (-38) >> >> >> >> Brian Andrus >> >> -Original Message- >> From: lustre-discuss-boun...@lists.lustre.org >> [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrus, >> Brian Contractor >> Sent: Wednesday, December 15, 2010 11:48 PM >> To: Andreas Dilger >> Cc: lustre-discuss@lists.lustre.org >> Subject: Re: [Lustre-discuss] Unable to mount OSTs >> >> I actually did run writeconf on the MGS/MDS as well. >> I just did it again to test. >> >> Followed the instructions in section 4.3.11 in the luster manual. >> Mounting MDT goes fine. >> Mounting the OSTs, the first just seems to be hanging. The other two >> both gave: >> [r...@ost2 ~]# mount -t lustre /dev/VG_lustre/scratch_ost2 >> /mnt/lustre/scratch/ost2/ >> mount.lustre: mount /dev/VG_lustre/scratch_ost2 at >> /mnt/lustre/scratch/ost2 failed: Function not implemented >> >> Currently running e2fsck on the two OSTs that didn't want to mount. I >> have already run it, but doing so again, just in case. >> >> Brian Andrus >> >> -Original Message- >> From: Andreas Dilger [mailto:andreas.dil...@oracle.com] >> Sent: Wednesday, December 15, 2010 10:55 PM >> To: Andrus, Brian Contractor >> Cc: lustre-discuss@lists.lustre.org >> Subject: Re: [Lustre-discuss] Unable to mount OSTs >> >> On 2010-12-15, at 22:50, Andrus, Brian Contractor wrote: >>> I did change IPs of the NID for 2 or our lustre filesystems. >>> Made appropriate changes to each ost. One thing I did that I probably >> shouldn't have was doing a --writeconf on the OSTs. Now I see: >>> >>> Flags: 0x142 >>> (OST update writeconf ) >>> and I cannot mount them. I get: >>> >>> mount.lustre: mount /dev/VG_lustre/scratch_ost1 at >> /mnt/lustre/scratch/ost1 failed: Function not implemented >>> however I do seem to be able to mount the OSTs that I didn't do >> --writeconf on. >> >> You can binary edit this flag in the "CONFIGS/mountdata" file on each >> OST. >> Alternately, you could follow the process in the manual, and run >> writeconf on the MGS and MDS as well, and then the OSTs should be able >> to register again. >> >>> I see in syslog: >>> LustreError: 6182:0:(obd_mount.c:1097:server_start_targets()) Required >> registration failed for nscratch-OST0001: -38 >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> ___ >> Lustre-discuss mailing list >> Lustre-discuss@lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Unable to mount OSTs
Thank you for your assistance. I am almost there. I am able to get all the OSTs to mount, but when I do "lctl dl", one of them is not showing up for the work filesystem. I can ping all OSSs from each other and once I have done that see them all in a peer_list. I have even tried mounting the OST on an OSS that already has another OST that is being recognized. Brian Andrus -Original Message- From: Wang Yibin [mailto:wang.yi...@oracle.com] Sent: Thursday, December 16, 2010 12:29 AM To: Andrus, Brian Contractor Cc: Andreas Dilger; lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Unable to mount OSTs This error sounds like that there is some unrecognized parameter(s) on your OSTs. Please check your OST parameters. You hit this issue mostly because we have some fix disallowing unrecognized parameters (22386). 在 2010-12-16，下午3:56， Andrus, Brian Contractor 写道： > Well e2fsck didn't help. Same error for those two (Function not > implemented). > I also see in the MGS/MDS log each time I try: > LustreError: 5974:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed > to write nscratch-OST0001 log (-38) > LustreError: 5973:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed > to write nscratch-OST0002 log (-38) > > > > Brian Andrus > > -Original Message- > From: lustre-discuss-boun...@lists.lustre.org > [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrus, > Brian Contractor > Sent: Wednesday, December 15, 2010 11:48 PM > To: Andreas Dilger > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > I actually did run writeconf on the MGS/MDS as well. > I just did it again to test. > > Followed the instructions in section 4.3.11 in the luster manual. > Mounting MDT goes fine. > Mounting the OSTs, the first just seems to be hanging. The other two > both gave: > [r...@ost2 ~]# mount -t lustre /dev/VG_lustre/scratch_ost2 > /mnt/lustre/scratch/ost2/ > mount.lustre: mount /dev/VG_lustre/scratch_ost2 at > /mnt/lustre/scratch/ost2 failed: Function not implemented > > Currently running e2fsck on the two OSTs that didn't want to mount. I > have already run it, but doing so again, just in case. > > Brian Andrus > > -Original Message- > From: Andreas Dilger [mailto:andreas.dil...@oracle.com] > Sent: Wednesday, December 15, 2010 10:55 PM > To: Andrus, Brian Contractor > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > On 2010-12-15, at 22:50, Andrus, Brian Contractor wrote: >> I did change IPs of the NID for 2 or our lustre filesystems. >> Made appropriate changes to each ost. One thing I did that I probably > shouldn't have was doing a --writeconf on the OSTs. Now I see: >> >> Flags: 0x142 >> (OST update writeconf ) >> and I cannot mount them. I get: >> >> mount.lustre: mount /dev/VG_lustre/scratch_ost1 at > /mnt/lustre/scratch/ost1 failed: Function not implemented >> however I do seem to be able to mount the OSTs that I didn't do > --writeconf on. > > You can binary edit this flag in the "CONFIGS/mountdata" file on each > OST. > Alternately, you could follow the process in the manual, and run > writeconf on the MGS and MDS as well, and then the OSTs should be able > to register again. > >> I see in syslog: >> LustreError: 6182:0:(obd_mount.c:1097:server_start_targets()) Required > registration failed for nscratch-OST0001: -38 > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Rebuilding kernel rpms
Hello, I need PAE-enabled kernels for my cluster. Thus, I would like to rebuild the kernel packages as provided by lustre for RHEL/CentOS. I tried to follow the guide on the Lustre wiki [1], which unfortunately just produces a very generic rpm which does not take care of initrd, grub etc. Additionally, the -devel package is missing, breaking many dependencies. I tried to integrate the method described in [1] with the guide for CentOS kernel compilatiom [2], but didn't succeed. Somehow, the lustre packages for ldiskfs aren't compiled automatically, and I get a lot of missing symbol errors upon installing the resulting rpms. What is the correct way of producing these kernel rpms? Cheers, Arne [1] http://wiki.lustre.org/index.php/Building_and_Installing_Lustre_from_Source_Code [2] http://wiki.centos.org/HowTos/Custom_Kernel -- Arne Brutschy Ph.D. StudentEmailarne.brutschy(AT)ulb.ac.be IRIDIA CP 194/6 Web iridia.ulb.ac.be/~abrutschy Universite' Libre de Bruxelles Tel +32 2 650 2273 Avenue Franklin Roosevelt 50 Fax +32 2 650 2715 1050 Bruxelles, Belgium (Fax at IRIDIA secretary) ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Lustre community build server
Hi Robert, That's very interesting. At CEA we also have a Hudson platform and I'm running acceptance-small for several Lustre branches in it. Hudson is a great tool but it was not design to test tools that run kernel-space that can crash your nodes or, at least, put your kernel in a bad shape. I will be very interested to share Hudson experience testing Lustre and see how you've configured it for your own tests. Aurélien Robert Read a écrit : > Hi, > > As I mentioned the other day, Whamcloud is hosting a Hudson build server and > producing snapshot builds for Centos 5.x (and Ubuntu 10.4 when it works) for > both 1.8.x and 2.x branches. Our intention is for this to be a resource for > the Lustre community to find recent Lustre packages for variety of Linux > distributions. Early next year we'll connect this to our test system so at > least some of the packages can be tested, as well. > > We would be interested in hearing from anyone that would like to participate > producing builds. Hudson is an distributed system, and it's easy to add more > build nodes, even behind firewalls (some of us are running build VMs on our > home machines). If you would like add another distribution or architecture we > don't have yet, or even one we do have (the more the merrier), we'd be happy > to work with you to do that. Please contact me if you are interested. > > cheers, > robert > > > > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] Unable to mount OSTs
This error sounds like that there is some unrecognized parameter(s) on your OSTs. Please check your OST parameters. You hit this issue mostly because we have some fix disallowing unrecognized parameters (22386). 在 2010-12-16，下午3:56， Andrus, Brian Contractor 写道： > Well e2fsck didn't help. Same error for those two (Function not > implemented). > I also see in the MGS/MDS log each time I try: > LustreError: 5974:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed > to write nscratch-OST0001 log (-38) > LustreError: 5973:0:(mgs_handler.c:487:mgs_handle_target_reg()) Failed > to write nscratch-OST0002 log (-38) > > > > Brian Andrus > > -Original Message- > From: lustre-discuss-boun...@lists.lustre.org > [mailto:lustre-discuss-boun...@lists.lustre.org] On Behalf Of Andrus, > Brian Contractor > Sent: Wednesday, December 15, 2010 11:48 PM > To: Andreas Dilger > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > I actually did run writeconf on the MGS/MDS as well. > I just did it again to test. > > Followed the instructions in section 4.3.11 in the luster manual. > Mounting MDT goes fine. > Mounting the OSTs, the first just seems to be hanging. The other two > both gave: > [r...@ost2 ~]# mount -t lustre /dev/VG_lustre/scratch_ost2 > /mnt/lustre/scratch/ost2/ > mount.lustre: mount /dev/VG_lustre/scratch_ost2 at > /mnt/lustre/scratch/ost2 failed: Function not implemented > > Currently running e2fsck on the two OSTs that didn't want to mount. I > have already run it, but doing so again, just in case. > > Brian Andrus > > -Original Message- > From: Andreas Dilger [mailto:andreas.dil...@oracle.com] > Sent: Wednesday, December 15, 2010 10:55 PM > To: Andrus, Brian Contractor > Cc: lustre-discuss@lists.lustre.org > Subject: Re: [Lustre-discuss] Unable to mount OSTs > > On 2010-12-15, at 22:50, Andrus, Brian Contractor wrote: >> I did change IPs of the NID for 2 or our lustre filesystems. >> Made appropriate changes to each ost. One thing I did that I probably > shouldn't have was doing a --writeconf on the OSTs. Now I see: >> >> Flags: 0x142 >> (OST update writeconf ) >> and I cannot mount them. I get: >> >> mount.lustre: mount /dev/VG_lustre/scratch_ost1 at > /mnt/lustre/scratch/ost1 failed: Function not implemented >> however I do seem to be able to mount the OSTs that I didn't do > --writeconf on. > > You can binary edit this flag in the "CONFIGS/mountdata" file on each > OST. > Alternately, you could follow the process in the manual, and run > writeconf on the MGS and MDS as well, and then the OSTs should be able > to register again. > >> I see in syslog: >> LustreError: 6182:0:(obd_mount.c:1097:server_start_targets()) Required > registration failed for nscratch-OST0001: -38 > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > ___ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
