RE: [CentOS] ext3 filesystems larger than 8TB
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eduardo Silvestre Sent: Sunday, May 04, 2008 9:04 PM To: CentOS mailing list Subject: Re: [CentOS] ext3 filesystems larger than 8TB But... Can i do that just with centos install cd and 3ware drivers? Format the volumes with 8TB/9TB/10TB without problems? Regards, --- Eduardo Silvestre nfsi telecom, lda. [EMAIL PROTECTED] Tel. (+351) 21 949 2300 - Fax (+351) 21 949 2301 http://www.nfsi.pt/ - Original Message - From: Ray Van Dolson [EMAIL PROTECTED] To: centos@centos.org Sent: Sexta-feira, 2 de Maio de 2008 21H15m GMT +00:00 GMT Britain, Ireland, Portugal Subject: Re: [CentOS] ext3 filesystems larger than 8TB On Fri, May 02, 2008 at 04:09:48PM -0400, Ross S. W. Walker wrote: I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross It's almost all Oracle database dump files... one database by itself is 4.5TB! Don't ask me what's in these things... :) Ray -- Using Oracle you are better of using XFS or RAW (raw meaning a drive partitioned with no filesystem). RAW would be better performance wise also. Or using RAW on a fast SAN network will work also. Hope that machine has plenty of RAM!!! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
I believe Florin Andrei had a typo in his message. No other file system will be as reliable as *XFS*. I've had XFS recover from system failures that Ext3 would/could not recover from. If you want dependability, reliability, and also large file systems, only use XFS. You'll find none better. You should also be using the 64 bit kernels as well, but then, you want that for a whole lot other reasons anyways. -- Brent L. Bates (UNIX Sys. Admin.) M.S. 912 Phone:(757) 865-1400, x204 NASA Langley Research CenterFAX:(757) 865-8177 Hampton, Virginia 23681-0001 Email: [EMAIL PROTECTED] http://www.vigyan.com/~blbates/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Ross S. W. Walker wrote: Ray Van Dolson wrote: On Fri, May 02, 2008 at 02:36:41PM -0500, Monty Shinn wrote: Greetings. I am trying to create a 10TB (approx) ext3 filesystem. I am able to successfully create the partition using parted, but when I try to use mkfs.ext3, I get an error stating there is an 8TB limit for ext3 filesystems. I looked at the specs for 5 on the upstream vendor's website, and they indicate that there is a 16TB limit on ext3. Has anyone been able to create a ext3 filesystem larger than 8TB? If ext3 isn't an option, has anyone used the kmod-xfs-smp.i686 module mentioned on the centos site? Surely it doesn't have an 8TB limit... specs: Centos 5.1 kernel-2.6.18-53.1.14.el5 3Ware 9550SXU 12port raid card Am I just walking into a big nightmare? Any thoughts would be greatly appreciated. I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross __ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Ross, We basically store video image sequences (edited and source) and audio/video files on our servers. We are an editing and broadcast design facility, doing mostly HD work. The files are relatively large, and there are a lot of them. I am trying to max out our current server population, moving from 250 and 500 gig drives to the Seagate 1TB enterprise (ES.2) SATA drives using the 12 port 3ware raid card. I have at least 4 servers that I am wanting to upgrade this way. They're just file servers running NFS and Samba. Do I *need* a 10TB partition? No, not really. I could segment into 2 5TB partitions if needed, and I may still end up doing that. I am beginning to wonder if the 8TB ext3 limit has been vetted enough. It is just easier for the users if it was one partition. I have to say when the mkfs.ext3 code hasn't been changed to allow 8TB partitions without adding the -F, (which did seem to work) it gives me pause. Naively perhaps, I didn't think it would be an issue. Thanks, Monty ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Mon, May 5, 2008 at 7:30 AM, Brent L. Bates [EMAIL PROTECTED] wrote: I believe Florin Andrei had a typo in his message. No other file system will be as reliable as *XFS*. I've had XFS recover from system failures that Ext3 would/could not recover from. If you want dependability, reliability, and also large file systems, only use XFS. Have you also had emacs write files where vim would not? Have you had qmail deliver where postfix would not? I mean, if you're going to attempt to start a holy war here in the mailing list, why stop at just filesystems :-P I believe XFS has some very good points where ext3 definitely lacks. That said, on RHEL/CentOS with the 4k stack compilation, xfs on x86 systems where you were using an abstraction layer (lvm, software raid, etc) xfs could get angry with you pretty quickly. EXT3 is widely regarded as being more stable than others over the long-term, which is why it's the default for a number of distros. Personal experiences may vary. -- During times of universal deceit, telling the truth becomes a revolutionary act. George Orwell ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Mon, May 5, 2008 at 12:51 PM, Monty Shinn [EMAIL PROTECTED] wrote: We basically store video image sequences (edited and source) and audio/video files on our servers. We are an editing and broadcast design facility, doing mostly HD work. The files are relatively large, and there are a lot of them. Having worked for a large film and television post-production facility in London for just over six years, XFS has been the primary filesystem for all our servers. Much of the data was split across multiple disk servers - each with around 2-3Tb of data. The whole filesystem was presented to the workstations over NFS with scripts to manage links to the different file servers - presenting a unified filesystem to the artist. XFS had given us the performance and reliability required and has gotten us out of some nasty scrapes. It's a shame that RHEL/CentOS does not include XFS as a choice of filesystem out of the box without having to compile the XFS module or use CentOS Extras repository, but perhaps one day it may happen.. ;) M. -- Martyn Drake http://www.drake.org.uk http://www.mindthegapps.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
RE: [CentOS] ext3 filesystems larger than 8TB
Monty Shinn wrote: Ross, We basically store video image sequences (edited and source) and audio/video files on our servers. We are an editing and broadcast design facility, doing mostly HD work. The files are relatively large, and there are a lot of them. I am trying to max out our current server population, moving from 250 and 500 gig drives to the Seagate 1TB enterprise (ES.2) SATA drives using the 12 port 3ware raid card. I have at least 4 servers that I am wanting to upgrade this way. They're just file servers running NFS and Samba. Do I *need* a 10TB partition? No, not really. I could segment into 2 5TB partitions if needed, and I may still end up doing that. I am beginning to wonder if the 8TB ext3 limit has been vetted enough. It is just easier for the users if it was one partition. I have to say when the mkfs.ext3 code hasn't been changed to allow 8TB partitions without adding the -F, (which did seem to work) it gives me pause. Naively perhaps, I didn't think it would be an issue. Makes sense, does NFS support sharing such large volumes? I suppose that will depend on both the server version of NFS and the client, but it's something you need to keep in mind. I think for a large file file system xfs is probably what you want, but you will want to run CentOS 64-bit with the 8k stacks to see it's full robustness and stability. Some people think xfs is good everywhere, but that's simply not true, I always recommend putting the OS on ext3 and then choosing the file system for your application data that best suits the application. Basically you have ext3, jfs, xfs, gfs and ocfs, the last 2 being clustered file systems. ext3 is good because it is widely supported and performs well under mixed work load, jfs is supposedly excellent if you have a lot of small files like a mail/news server, xfs for large files and of course gfs or ocfs for clusters that need simultaneous file system access from multiple nodes (but they are slower due to locking overhead). If you have volumes over 8TB then you really need to use either jfs or xfs depending on the application and if you are using xfs I highly recommend you run 64-bit for stability reasons. -Ross __ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Martyn Drake wrote: Having worked for a large film and television post-production facility in London for just over six years, XFS has been the primary filesystem for all our servers. Much of the data was split across multiple disk servers - each with around 2-3Tb of data. The whole filesystem was presented to the workstations over NFS with scripts to manage links to the different file servers - presenting a unified filesystem to the artist. XFS had given us the performance and reliability required and has gotten us out of some nasty scrapes. Well, XFS was designed exactly for the kind of scenario you're describing. No wonder it performs really well in that sort of situation. -- Florin Andrei http://florin.myip.org/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Florin Andrei wrote: Martyn Drake wrote: Having worked for a large film and television post-production facility in London for just over six years, XFS has been the primary filesystem for all our servers. Much of the data was split across multiple disk servers - each with around 2-3Tb of data. The whole filesystem was presented to the workstations over NFS with scripts to manage links to the different file servers - presenting a unified filesystem to the artist. XFS had given us the performance and reliability required and has gotten us out of some nasty scrapes. Well, XFS was designed exactly for the kind of scenario you're describing. No wonder it performs really well in that sort of situation. Thanks to all for your help. I decided to go with XFS, since it could be loaded as a module and I have worked with it on SGI platforms for years. Thanks again. Monty ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Jim Perrin wrote: Personal experiences may vary. Yup. Do your own tests, involving your particular situation, then draw conclusions. The average may just not apply very well in your case. -- Florin Andrei http://florin.myip.org/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Ross S. W. Walker wrote: jfs is supposedly excellent if you have a lot of small files like a mail/news server Hm, last time I tested ReiseFS turned out to be the best FS for that situation. But it's been a while, perhaps things have changed a bit. -- Florin Andrei http://florin.myip.org/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
RE: [CentOS] ext3 filesystems larger than 8TB
Florin Andrei wrote: Ross S. W. Walker wrote: jfs is supposedly excellent if you have a lot of small files like a mail/news server Hm, last time I tested ReiseFS turned out to be the best FS for that situation. But it's been a while, perhaps things have changed a bit. Yeah, but reiserfs is all but dead these days. At least I wouldn't plan a long-term deployment around it... -Ross __ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Mon, May 05, 2008, Florin Andrei wrote: Ross S. W. Walker wrote: jfs is supposedly excellent if you have a lot of small files like a mail/news server Hm, last time I tested ReiseFS turned out to be the best FS for that situation. But it's been a while, perhaps things have changed a bit. Best only if one ignores the reiserfs' tendency to trash data, particularly on abnormal shutdowns. Bill -- INTERNET: [EMAIL PROTECTED] Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way Voice: (206) 236-1676 Mercer Island, WA 98040-0820 Fax:(206) 232-9186 Ah, you know the type. They like to blame it all on the Jews or the Blacks, 'cause if they couldn't, they'd have to wake up to the fact that life's one big, scary, glorious, complex and ultimately unfathomable crapshoot -- and the only reason THEY can't seem to keep up is they're a bunch of misfits and losers. -- A analysis of Neo-Nazis, from The Badger comic ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
But... Can i do that just with centos install cd and 3ware drivers? Format the volumes with 8TB/9TB/10TB without problems? Regards, --- Eduardo Silvestre nfsi telecom, lda. [EMAIL PROTECTED] Tel. (+351) 21 949 2300 - Fax (+351) 21 949 2301 http://www.nfsi.pt/ - Original Message - From: Ray Van Dolson [EMAIL PROTECTED] To: centos@centos.org Sent: Sexta-feira, 2 de Maio de 2008 21H15m GMT +00:00 GMT Britain, Ireland, Portugal Subject: Re: [CentOS] ext3 filesystems larger than 8TB On Fri, May 02, 2008 at 04:09:48PM -0400, Ross S. W. Walker wrote: I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross It's almost all Oracle database dump files... one database by itself is 4.5TB! Don't ask me what's in these things... :) Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Fri, 2 May 2008 at 2:36pm, Monty Shinn wrote I am trying to create a 10TB (approx) ext3 filesystem. I am able to successfully create the partition using parted, but when I try to use mkfs.ext3, I get an error stating there is an 8TB limit for ext3 filesystems. I looked at the specs for 5 on the upstream vendor's website, and they indicate that there is a 16TB limit on ext3. Has anyone been able to create a ext3 filesystem larger than 8TB? Yes. 'mke2fs -F' forces it to make the FS, even though it thinks it's too big. They should have changed that when 16TB ext3 fs support moved from tech preview to production ready, but I think it got missed. Maybe in 5.2... -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Fri, May 02, 2008 at 02:36:41PM -0500, Monty Shinn wrote: Greetings. I am trying to create a 10TB (approx) ext3 filesystem. I am able to successfully create the partition using parted, but when I try to use mkfs.ext3, I get an error stating there is an 8TB limit for ext3 filesystems. I looked at the specs for 5 on the upstream vendor's website, and they indicate that there is a 16TB limit on ext3. Has anyone been able to create a ext3 filesystem larger than 8TB? If ext3 isn't an option, has anyone used the kmod-xfs-smp.i686 module mentioned on the centos site? Surely it doesn't have an 8TB limit... specs: Centos 5.1 kernel-2.6.18-53.1.14.el5 3Ware 9550SXU 12port raid card Am I just walking into a big nightmare? Any thoughts would be greatly appreciated. I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
RE: [CentOS] ext3 filesystems larger than 8TB
Ray Van Dolson wrote: On Fri, May 02, 2008 at 02:36:41PM -0500, Monty Shinn wrote: Greetings. I am trying to create a 10TB (approx) ext3 filesystem. I am able to successfully create the partition using parted, but when I try to use mkfs.ext3, I get an error stating there is an 8TB limit for ext3 filesystems. I looked at the specs for 5 on the upstream vendor's website, and they indicate that there is a 16TB limit on ext3. Has anyone been able to create a ext3 filesystem larger than 8TB? If ext3 isn't an option, has anyone used the kmod-xfs-smp.i686 module mentioned on the centos site? Surely it doesn't have an 8TB limit... specs: Centos 5.1 kernel-2.6.18-53.1.14.el5 3Ware 9550SXU 12port raid card Am I just walking into a big nightmare? Any thoughts would be greatly appreciated. I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross __ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Fri, May 02, 2008 at 04:09:48PM -0400, Ross S. W. Walker wrote: I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross It's almost all Oracle database dump files... one database by itself is 4.5TB! Don't ask me what's in these things... :) Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
On Fri, May 02, 2008 at 04:40:21PM -0400, Ross S. W. Walker wrote: Ray Van Dolson wrote: On Fri, May 02, 2008 at 04:09:48PM -0400, Ross S. W. Walker wrote: I think it *theoretically* should work as ext3 should to 16TB. However, we ran into issues with userland tools and such. Possibly related to x86_64 vs i386 stuff, but in the end to avoid continued troubleshooting we just used centosplus + jfs. Works perfectly for our 10TB filesystem. I'm curious what you store that you need 10TB of linear storage? I have had 4-6-8TB storage systems, but the storage was always divvied up between different applications. -Ross It's almost all Oracle database dump files... one database by itself is 4.5TB! Don't ask me what's in these things... :) Ah, should have figured it was for backup. 4.5TB, wow, sounds like financial forecasting or geographic survey data... The latter I'm sure. I work at ESRI :) Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] ext3 filesystems larger than 8TB
Monty Shinn wrote: I am trying to create a 10TB (approx) ext3 filesystem. I am able to successfully create the partition using parted, but when I try to use mkfs.ext3, I get an error stating there is an 8TB limit for ext3 filesystems. If ext3 isn't an option, has anyone used the kmod-xfs-smp.i686 module mentioned on the centos site? Surely it doesn't have an 8TB limit... You'll have all sorts of troubles with Ext3 once you step deeper into the TB zone. Just use XFS. That's what we did, works fine for us. And it's typically faster when the files are very large (especially delete is very fast). No other filesystem will be as reliable as Ext3 if the machine suddenly loses power, but if you have a battery backup or something like that, you should be fine with non-Ext3 filesystems. -- Florin Andrei http://florin.myip.org/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
