Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thursday, April 14, 2011 11:19 PM, Ray Van Dolson wrote: On Thu, Apr 14, 2011 at 10:44:00PM +0800, Christopher Chan wrote: On Thursday, April 14, 2011 10:11 PM, Ray Van Dolson wrote: On Thu, Apr 14, 2011 at 10:07:55PM +0800, Emmanuel Noobadmin wrote: On 4/14/11, John R Piercepie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? ZFS and OCFS play in different spaces. And ZFS is going nowhere... if you want to use on an open OS, OpenIndiana may be a good bet, but you're best short-term / mature option would be Nexenta or Solaris Express. Huh? What gives Nexenta a better advantage over OpenIndiana? They are both in the same boat. Both will have to migrate to illumos and move away from the last OpenSolaris ON release. Oh, Nexenta has a company backing it? Makes no different when both projects will be using the same core image. Now, if OpenIndiana resists using illumos, then you will have a case for Nexenta over OpenIndiana. OpenIndiana is in their what, first release? I don't think that Nexenta 3.x is based on it *yet*. Both will eventually converge. In the meantime, yes, for storage needs I'd go with Nexenta for the reasons you mentioned. :) Hardy userland, gcc compiled and gnu linked...hmm...I'll give Nexenta a shot after they start basing on perhaps Lucid repos. For personal use? Maybe different factors. Nexenta the company of course will be contributing to OpenIndiana and Illumos... Now that is news to me. I know that Garrett would be willing to spare a man IF the OpenIndiana guys start using illumos as their base for the next release... ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On 04/14/11 5:43 PM, Christopher Chan wrote: On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. afaik, both are still using pretty much the last opensolaris kernel with minor changes I was going on this, which says OpenIndiana is a member of the Illumos Foundation, that Illumos was providing the core/kernel, and OpenIndiana is integrating it into a complete system aka distribution http://wiki.openindiana.org/oi/Frequently+Asked+Questions#FrequentlyAskedQuestions-WhatistherelationshipbetweenOpenIndianaandIllumos%3F They go onto say they are waiting for Illumos to mature before they integrate it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
John R Pierce wrote: On 04/14/11 5:43 PM, Christopher Chan wrote: On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. afaik, both are still using pretty much the last opensolaris kernel with minor changes I was going on this, which says OpenIndiana is a member of the Illumos Foundation, that Illumos was providing the core/kernel, and OpenIndiana is integrating it into a complete system aka distribution http://wiki.openindiana.org/oi/Frequently+Asked+Questions#FrequentlyAskedQuestions-WhatistherelationshipbetweenOpenIndianaandIllumos%3F They go onto say they are waiting for Illumos to mature before they integrate it. Eham..., CentOS mailinglist maybe to continue in private? Ljubomir ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Fri, Apr 15, 2011 at 10:40 AM, Ljubomir Ljubojevic off...@plnet.rswrote: John R Pierce wrote: On 04/14/11 5:43 PM, Christopher Chan wrote: On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. afaik, both are still using pretty much the last opensolaris kernel with minor changes I was going on this, which says OpenIndiana is a member of the Illumos Foundation, that Illumos was providing the core/kernel, and OpenIndiana is integrating it into a complete system aka distribution http://wiki.openindiana.org/oi/Frequently+Asked+Questions#FrequentlyAskedQuestions-WhatistherelationshipbetweenOpenIndianaandIllumos%3F They go onto say they are waiting for Illumos to mature before they integrate it. Eham..., CentOS mailinglist maybe to continue in private? Ljubomir Eham., many people are learning a lot more from this thread than a lot of the other threads in the past few days. let them continue, and don't subscribe to the tread :) -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Friday, April 15, 2011 03:59 PM, John R Pierce wrote: On 04/14/11 5:43 PM, Christopher Chan wrote: On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. afaik, both are still using pretty much the last opensolaris kernel with minor changes or nice big changes from the standpoint of those who were pining for openindiana with b134+patches I was going on this, which says OpenIndiana is a member of the Illumos Foundation, that Illumos was providing the core/kernel, and OpenIndiana is integrating it into a complete system aka distribution http://wiki.openindiana.org/oi/Frequently+Asked+Questions#FrequentlyAskedQuestions-WhatistherelationshipbetweenOpenIndianaandIllumos%3F oh i see. They go onto say they are waiting for Illumos to mature before they integrate it. Yes...like getting g11n in. I guess traction is there already. OpenIndiana will be moving to illumos so i guess it would be the one to use if one wants a sun cc compiled and sun linked distro. It's going to be interesting to see how all these different projects including CentOS play out. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Apr 15, 2011, at 4:48 AM, Rudi Ahlers r...@softdux.com wrote: On Fri, Apr 15, 2011 at 10:40 AM, Ljubomir Ljubojevic off...@plnet.rs wrote: John R Pierce wrote: On 04/14/11 5:43 PM, Christopher Chan wrote: On Friday, April 15, 2011 02:46 AM, John R Pierce wrote: On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. Eh? I was under the impression that they are separate and that Garrett Damore was rather unhappy with the initial direction of OpenIndiana in not preparing for an illumos release. 148 is still not illumos as far as I know. afaik, both are still using pretty much the last opensolaris kernel with minor changes I was going on this, which says OpenIndiana is a member of the Illumos Foundation, that Illumos was providing the core/kernel, and OpenIndiana is integrating it into a complete system aka distribution http://wiki.openindiana.org/oi/Frequently+Asked+Questions#FrequentlyAskedQuestions-WhatistherelationshipbetweenOpenIndianaandIllumos%3F They go onto say they are waiting for Illumos to mature before they integrate it. Eham..., CentOS mailinglist maybe to continue in private? Ljubomir Eham., many people are learning a lot more from this thread than a lot of the other threads in the past few days. let them continue, and don't subscribe to the tread :) I agree with both assessments, but since this is a CentOS list and this thread has now twisted into ZFS advocacy I must say as well, continue off list. -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thursday, April 14, 2011 01:51 PM, John R Pierce wrote: On 04/13/11 9:51 PM, Emmanuel Noobadmin wrote: I'm was stuck trying to decide whether to go for the cheaper RAID 5 setup and possibly getting killed by the IOPS penalty and the risk associated with rebuild time, or figure out a way to use the recommended RAID 10 setup with a smaller usable capacity for the budget but do so with the ability to expand in the near future. So really hoping that it could be done. since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. Special cases warrant exemption. I, too, run OpenIndiana... Use the right tool for the job. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On 4/14/11, John R Pierce pie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thu, Apr 14, 2011 at 10:07:55PM +0800, Emmanuel Noobadmin wrote: On 4/14/11, John R Pierce pie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? ZFS and OCFS play in different spaces. And ZFS is going nowhere... if you want to use on an open OS, OpenIndiana may be a good bet, but you're best short-term / mature option would be Nexenta or Solaris Express. Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thursday, April 14, 2011 10:11 PM, Ray Van Dolson wrote: On Thu, Apr 14, 2011 at 10:07:55PM +0800, Emmanuel Noobadmin wrote: On 4/14/11, John R Piercepie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? ZFS and OCFS play in different spaces. And ZFS is going nowhere... if you want to use on an open OS, OpenIndiana may be a good bet, but you're best short-term / mature option would be Nexenta or Solaris Express. Huh? What gives Nexenta a better advantage over OpenIndiana? They are both in the same boat. Both will have to migrate to illumos and move away from the last OpenSolaris ON release. Oh, Nexenta has a company backing it? Makes no different when both projects will be using the same core image. Now, if OpenIndiana resists using illumos, then you will have a case for Nexenta over OpenIndiana. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thursday, April 14, 2011 10:07 PM, Emmanuel Noobadmin wrote: On 4/14/11, John R Piercepie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. /me is so happy that the Indiana project surfaced. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? I have found ZFS to be very much stable on OpenSolaris stable releases and have not had any issues on OpenIndiana 147 with 9 1TB disks in a raidz2 pool. There are reports of those who had problems when upgrading but I however have not - whether difference in env/procedure was responsible I do not know but those chaps got help on the Openindiana ml with their zfs problems. I have used OpenSolaris since 2008.05 stable release without issues. Long term support/updates is not an absolute certainty at the moment with regards to OpenIndiana but that is only in the terms of packages outside the core image since illumos is guaranteed. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Thu, Apr 14, 2011 at 10:44:00PM +0800, Christopher Chan wrote: On Thursday, April 14, 2011 10:11 PM, Ray Van Dolson wrote: On Thu, Apr 14, 2011 at 10:07:55PM +0800, Emmanuel Noobadmin wrote: On 4/14/11, John R Piercepie...@hogranch.com wrote: since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes I was actually considering this but then came news that Oracle was killing OpenSolaris and likely to be pushing OCFS so decided I probably don't want to have something come bite me a year or two down the road. I'm not sure how things developed since then though. But based on your recommendation and Christopher Chan's, it would seem like you guys don't think that long term support/updates would be an issue for ZFS? ZFS and OCFS play in different spaces. And ZFS is going nowhere... if you want to use on an open OS, OpenIndiana may be a good bet, but you're best short-term / mature option would be Nexenta or Solaris Express. Huh? What gives Nexenta a better advantage over OpenIndiana? They are both in the same boat. Both will have to migrate to illumos and move away from the last OpenSolaris ON release. Oh, Nexenta has a company backing it? Makes no different when both projects will be using the same core image. Now, if OpenIndiana resists using illumos, then you will have a case for Nexenta over OpenIndiana. OpenIndiana is in their what, first release? I don't think that Nexenta 3.x is based on it *yet*. Both will eventually converge. In the meantime, yes, for storage needs I'd go with Nexenta for the reasons you mentioned. :) For personal use? Maybe different factors. Nexenta the company of course will be contributing to OpenIndiana and Illumos... Ray ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On 04/14/11 7:44 AM, Christopher Chan wrote: Now, if OpenIndiana resists using illumos... openindiana is under the Illumos project umbrella. They aren't going to use anything else. someone suggested Solaris Express, that has no patches or updates unless you subscribe to annual support at several $1000/year/CPU socket. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On Wed, Apr 13, 2011 at 11:19 AM, Emmanuel Noobadmin centos.ad...@gmail.com wrote: On 4/13/11, Rudi Ahlers r...@softdux.com wrote: to expand the array :) I haven't had problems doing it this way yet. I finally figured out my mistake creating the raid devices and got a working RAID 0 on two RAID 1 arrays. But I wasn't able to add another RAID 1 component to the array with the error mdadm: add new device failed for /dev/md/mdr1_3 as 2: Invalid argument Googling up this indicates that it's the expected result trying to add a new device to a RAID 0 array. Could you or anybody else please share what's the trick to achieving this? You can't expand a mdraid raid0. I believe you can expand a mdraid raid10,5,6, but not raid0. If you want to use separate raid1 devices instead of mdraid's raid10 implementation then use LVM, add them to a VG then stripe the LVs across the different PVs. The only problem with that is restriping existing LVs across new PVs is difficult to the point that it is often better to create new LVs with the proper striping. You can do it though using 'lvresize' to change the stripe width, but it won't give a linear striping for existing data and the LV will eventually fill the first PVs causing all data to only be written to the last PV. I often find it handy to have a backup raid1 disk on the system that's big enough to hold the contents of the largest LV, then dump the production LV to the backup, blow away the production, recreate with the new stripe size, then restore the data back. This backup volume could be an iSCSI volume exported from another server that does have the capacity if there isn't any in the host. -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On 4/14/11, Ross Walker rswwal...@gmail.com wrote: You can't expand a mdraid raid0. I believe you can expand a mdraid raid10,5,6, but not raid0. That was what I thought previously when looking into this and weighing the pros/cons of using RAID 10 vs RAID 5. But earlier this week, from the 40TB Filesystem thread, Rudi stated that he has a RAID 0 on RAID 1 setup that he can expand and Brandon confirmed that it is possible to expand RAID 0. http://forum.nginx.org/read.php?24,190407,190764#msg-190764 Since I've read that some features such as creating RAID 10 directly were not in the man pages, and another person in the mdadm list implied that a raid 10 like array could be achieved using some creative RAID 5 layout, I assumed that perhaps it was possible to do that, just that there was some undocumented trick or specific manner the arrays had to be setup. I'm was stuck trying to decide whether to go for the cheaper RAID 5 setup and possibly getting killed by the IOPS penalty and the risk associated with rebuild time, or figure out a way to use the recommended RAID 10 setup with a smaller usable capacity for the budget but do so with the ability to expand in the near future. So really hoping that it could be done. I often find it handy to have a backup raid1 disk on the system that's big enough to hold the contents of the largest LV, then dump the production LV to the backup, blow away the production, recreate with the new stripe size, then restore the data back. This backup volume could be an iSCSI volume exported from another server that does have the capacity if there isn't any in the host. That sounds like a solution since each LV shouldn't be too massive. This seems to imply I would have to bring down the service running off the LV during the recreation but I suppose I since the storage is planned to be exported over iSCSI in the first place, I could simply export the backup copy while expanding the original. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Expanding RAID 10 array, WAS: 40TB File System Recommendations
On 04/13/11 9:51 PM, Emmanuel Noobadmin wrote: I'm was stuck trying to decide whether to go for the cheaper RAID 5 setup and possibly getting killed by the IOPS penalty and the risk associated with rebuild time, or figure out a way to use the recommended RAID 10 setup with a smaller usable capacity for the budget but do so with the ability to expand in the near future. So really hoping that it could be done. since this is the centos list, I really didn't want to suggest this, but if I was building a 20 or 40TB or whatever storage server, I do believe I'd be strongly consider using Solaris, or one of its variants like OpenIndiana, with ZFS. ZFS was engineered from the ground up to scale to zetabytes # zpool create archive mirror c2t0d0 c2t1d0 mirror c2t2d0 c2t3d0 mirror c3t0d0 c3t1d0 mirror c3t2d0 c3t2d0 . spare c?t?d0 c?t?d0 done. available for use in a few seconds. default mountpoint is /archive # zfs create -o mountpoint=/u01 archive/u01 # zfs create -o mountpoint=/u02 archive/u02 creates a couple more filesystems that share the free space, mounted as /u01 and /u02 adding more disks? # zpool add archive mirror c7t0d0 c7t1d0 mirror c7t2d0 c7t3d0 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
