Re: very degraded RAID5, or increasing capacity by adding discs
Louis-David Mitterrand wrote: On Tue, Oct 09, 2007 at 01:48:50PM +0400, Michael Tokarev wrote: There still is - at least for ext[23]. Even offline resizers can't do resizes from any to any size, extfs developers recommend to recreate filesystem anyway if size changes significantly. I'm too lazy to find a reference now, it has been mentioned here on linux-raid at least this year. It's sorta like fat (yea, that ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, everything is ok, but if you want to go from 501Mb to 1Gb+1, you have to recreate almost all data structures because sizes of all internal fields changes - and here it's much safer to just re-create it from scratch than trying to modify it in place. Sure it's much better for extfs, but the point is still the same. I'll just mention that I once resized a multi-Tera ext3 filesystem and it took 8hours +, a comparable XFS online resize lasted all of 10 seconds! Because of the different way these file systems do things, there is no comparable resize, at least in terms of work to be done. For many systems R/W operations are more common than resize, so the F/S type is selected to optimize that. ;-) -- bill davidsen <[EMAIL PROTECTED]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
On Tue, Oct 09, 2007 at 01:48:50PM +0400, Michael Tokarev wrote: > > There still is - at least for ext[23]. Even offline resizers > can't do resizes from any to any size, extfs developers recommend > to recreate filesystem anyway if size changes significantly. > I'm too lazy to find a reference now, it has been mentioned here > on linux-raid at least this year. It's sorta like fat (yea, that > ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, > everything is ok, but if you want to go from 501Mb to 1Gb+1, you > have to recreate almost all data structures because sizes of > all internal fields changes - and here it's much safer to just > re-create it from scratch than trying to modify it in place. > Sure it's much better for extfs, but the point is still the same. I'll just mention that I once resized a multi-Tera ext3 filesystem and it took 8hours +, a comparable XFS online resize lasted all of 10 seconds! - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
On Tuesday October 9, [EMAIL PROTECTED] wrote: > Hello Neil , > > On Tue, 9 Oct 2007, Neil Brown wrote: > > On Tuesday October 9, [EMAIL PROTECTED] wrote: > >> > >> Problems at step 4.: 'man mdadm' doesn't tell if it's possible to > >> grow an array to a degraded array (non existant disc). Is it possible? > > > > Why not experiment with loop devices on files and find out? > > > > But yes: you can grow to a degraded array providing you specify a > > --backup-file. > Is there an estimate of how large this file can get ? > It's probably a calculation based on disk & array parameters . > But I was unable to find a reference on it from the manpage . > > Tia , JimL I think: chunk-size * old-ndisks * new-ndisks * 2 + 1K It might be smaller than that, but it shouldn't be larger. NeilBrown - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Hello Neil , On Tue, 9 Oct 2007, Neil Brown wrote: On Tuesday October 9, [EMAIL PROTECTED] wrote: Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible? Why not experiment with loop devices on files and find out? But yes: you can grow to a degraded array providing you specify a --backup-file. Is there an estimate of how large this file can get ? It's probably a calculation based on disk & array parameters . But I was unable to find a reference on it from the manpage . Tia , JimL However I don't recommend it. I would never recommend having a degraded array by design. It should only ever happen due to a failure, and should last only until you can get a replacement rebuilt. Remember that a degraded raid5 has a greater risk of data loss than a single drive. PS: the fact, that degraded array will be unsafe for the data is an intented motivating factor for buying next drive ;) :-) NeilBrown -- +-+ | James W. Laferriere | System Techniques | Give me VMS | | NetworkEngineer | 663 Beaumont Blvd | Give me Linux | | [EMAIL PROTECTED] | Pacifica, CA. 94044 | only on AXP | +-+ - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Michael Tokarev said: (by the date of Tue, 09 Oct 2007 02:52:06 +0400) > Janek Kozicki wrote: > > Hello, > > > > Recently I started to use mdadm and I'm very impressed by its > > capabilities. > > > > I have raid0 (250+250 GB) on my workstation. And I want to have > > raid5 (4*500 = 1500 GB) on my backup machine. > > Hmm. Are you sure you need that much space on the backup, to > start with? Maybe better backup strategy will help to avoid > hardware costs? Such as using rsync for backups as discussed > on this mailinglist about a month back (rsync is able to keep > many ready to use copies of your filesystems but only store > files that actually changed since the last backup, thus > requiring much less space than many full backups). yes, exactly. I am using rsnapshot, which is based on rsync and hardlinks. It works exceptionally well - to my knowledge it's the best backup solution I have ever seen. With plugin scripts I am even mounting an lvm-snapshot of the drive being backupped. from command 'rsnapshot du' I can see how many space is used (but each directory tree is a full backup (made with hardlinks)): 278G/backup/.sync 454M/backup/hourly.0/ 515M/backup/hourly.1/ 527M/backup/daily.0/ 30G /backup/daily.1/ 21G /backup/daily.2/ 561M/backup/daily.3/ 1.6G/backup/daily.4/ 3.0G/backup/daily.5/ 594M/backup/daily.6/ 1.4G/backup/weekly.0/ 11G /backup/weekly.1/ 9.3G/backup/weekly.2/ 23G /backup/weekly.3/ 33G /backup/monthly.0/ 3.7G/backup/monthly.1/ 415Gtotal > It's definitely not possible with raid5. Only option is to create a > raid5 array consisting of less drives than it should contain at the > end, and reshape it when you get more drives, as others noted in this > thread. But do note the following points: <..snip..> yes, I am aware of all those problems you listed. The data I'm talking about is already a backup. While the real data is on my workstation (a different linux box - albeit only the newest version of my data). Only losing both of them simultaneoulsy will be catastrophic for me. So I am inclined to do some experiments with the backup drives configuration, while still doing my best at not losing it. An exercise, you know :) > > is it just a pipe dream? > > I'd say it is... ;) oh well. But I learnt a lot from your answers, thanks a lot! PS: I'm receiving some mailing list posts twice, anybody knows why? I'm used to mailman but looks like majordomo is being configured in a different way - I cannot find a configure page. (I just subscribed). -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Neil Brown said: (by the date of Tue, 9 Oct 2007 13:32:09 +1000) > On Tuesday October 9, [EMAIL PROTECTED] wrote: > > > > Problems at step 4.: 'man mdadm' doesn't tell if it's possible to > > grow an array to a degraded array (non existant disc). Is it possible? > > Why not experiment with loop devices on files and find out? > > But yes: you can grow to a degraded array providing you specify a > --backup-file. Thanks! I'll test this on loopback devices :) -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Neil Brown wrote: > On Tuesday October 9, [EMAIL PROTECTED] wrote: [] >> o During this reshape time, errors may be fatal to the whole array - >> while mdadm do have a sense of "critical section", but the >> whole procedure isn't as much tested as the rest of raid code, >> I for one will not rely on it, at least for now. For example, >> a power failure at an "unexpected" moment, or some plain-stupid >> error in reshape code so that the whole array goes "boom" etc... > > While it is true that the resize code is less tested than other code, > it is designed to handle a single failure at any time (so a power > failure is OK as long as the array is not running degraded), and I > have said that if anyone does suffer problems while performing a > reshape, I will do my absolute best to get the array functioning and > the data safe again. Well... Neil, it's your code, so you trust it - that's ok, I also (tries to) trust my code until someone finds a bug in it.. ;) And I'm a sysadmin (among other things), who's professional property must be a bit of paranoia.. You got the idea ;) >> o A filesystem on the array has to be resized separately after >> re{siz,shap}ing the array. And filesystems are different at >> this point, too - there are various limitations. For example, >> it's problematic to grow ext[23]fs by large amounts, because >> when it gets initially created, mke2fs calculates sizes of >> certain internal data structures based on the device size, >> and those structures can't be grown significantly, only >> recreating the filesystem will do the trick. > > This isn't entirely true. > For online resizing (while the filesystem is mounted) there are some > limitations as you suggest. For offline resizing (while filesystem is > not mounted) there are no such limitations. There still is - at least for ext[23]. Even offline resizers can't do resizes from any to any size, extfs developers recommend to recreate filesystem anyway if size changes significantly. I'm too lazy to find a reference now, it has been mentioned here on linux-raid at least this year. It's sorta like fat (yea, that ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, everything is ok, but if you want to go from 501Mb to 1Gb+1, you have to recreate almost all data structures because sizes of all internal fields changes - and here it's much safer to just re-create it from scratch than trying to modify it in place. Sure it's much better for extfs, but the point is still the same. /mjt - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
On Tuesday October 9, [EMAIL PROTECTED] wrote: > > Problems at step 4.: 'man mdadm' doesn't tell if it's possible to > grow an array to a degraded array (non existant disc). Is it possible? Why not experiment with loop devices on files and find out? But yes: you can grow to a degraded array providing you specify a --backup-file. However I don't recommend it. I would never recommend having a degraded array by design. It should only ever happen due to a failure, and should last only until you can get a replacement rebuilt. Remember that a degraded raid5 has a greater risk of data loss than a single drive. > > > PS: the fact, that degraded array will be unsafe for the data is an > intented motivating factor for buying next drive ;) :-) NeilBrown - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
On Tuesday October 9, [EMAIL PROTECTED] wrote: > > o degraded raid5 isn't really Raid - i.e, it's not any better than > a raid0 array, that is, any disk fails => the whole array fails. > So instead of creating a degraded raid5 array initially, create > smaller one instead, but not degraded, and reshape it when > necessary. Fully agree. > > o reshaping takes time, and for this volume, reshape will take > many hours, maybe days, to complete. > > o During this reshape time, errors may be fatal to the whole array - > while mdadm do have a sense of "critical section", but the > whole procedure isn't as much tested as the rest of raid code, > I for one will not rely on it, at least for now. For example, > a power failure at an "unexpected" moment, or some plain-stupid > error in reshape code so that the whole array goes "boom" etc... While it is true that the resize code is less tested than other code, it is designed to handle a single failure at any time (so a power failure is OK as long as the array is not running degraded), and I have said that if anyone does suffer problems while performing a reshape, I will do my absolute best to get the array functioning and the data safe again. > > o A filesystem on the array has to be resized separately after > re{siz,shap}ing the array. And filesystems are different at > this point, too - there are various limitations. For example, > it's problematic to grow ext[23]fs by large amounts, because > when it gets initially created, mke2fs calculates sizes of > certain internal data structures based on the device size, > and those structures can't be grown significantly, only > recreating the filesystem will do the trick. This isn't entirely true. For online resizing (while the filesystem is mounted) there are some limitations as you suggest. For offline resizing (while filesystem is not mounted) there are no such limitations. NeilBrown - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: very degraded RAID5, or increasing capacity by adding discs
} -Original Message- } From: [EMAIL PROTECTED] [mailto:linux-raid- } [EMAIL PROTECTED] On Behalf Of Janek Kozicki } Sent: Monday, October 08, 2007 6:47 PM } To: linux-raid@vger.kernel.org } Subject: Re: very degraded RAID5, or increasing capacity by adding discs } } Janek Kozicki said: (by the date of Tue, 9 Oct 2007 00:25:50 +0200) } } > Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35 } +1300) } > } > > No, but you can make a degraded 3 drive array, containing 2 drives and } > > then add the next drive to complete it. } > > } > > The array can then be grown (man mdadm, GROW section), to add the } fourth. } > } > Oh, good. Thanks, I must've been blind that I missed this. } > This completely solves my problem. } } Uh, actually not :) } } My 1st 500 GB drive is full now. When I buy a 2nd one I want to } create a 3-disc degraded array using just 2 discs, one of which } contains unbackupable data. } } steps: } 1. create degraded two-disc RAID5 on 1 new disc } 2. copy data from old disc to new one } 3. rebuild the array with old and new discs (now I have 500 GB on 2 discs) 3. Add old disk to new array. Once done RAID5 is redundant. } 4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2 } discs) 4. Buy 3rd disk. 5. Add new 3rd disk to array and grow to 3 disk RAID5 array. Once done, array is redundant. Repeat 4 and 5 each time you buy a new disk. I don't think you can grow to a degraded array. I think you must add a new disk first. But I am not sure. } ... } 5. when I buy 3rd drive I either grow the array, or just rebuild and } wait with growing until I buy a 4th drive. } } Problems at step 4.: 'man mdadm' doesn't tell if it's possible to } grow an array to a degraded array (non existant disc). Is it possible? } } } PS: the fact, that degraded array will be unsafe for the data is an } intented motivating factor for buying next drive ;) } } -- } Janek Kozicki | } - } To unsubscribe from this list: send the line "unsubscribe linux-raid" in } the body of a message to [EMAIL PROTECTED] } More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Janek Kozicki wrote: > Hello, > > Recently I started to use mdadm and I'm very impressed by its > capabilities. > > I have raid0 (250+250 GB) on my workstation. And I want to have > raid5 (4*500 = 1500 GB) on my backup machine. Hmm. Are you sure you need that much space on the backup, to start with? Maybe better backup strategy will help to avoid hardware costs? Such as using rsync for backups as discussed on this mailinglist about a month back (rsync is able to keep many ready to use copies of your filesystems but only store files that actually changed since the last backup, thus requiring much less space than many full backups). > The backup machine currently doesn't have raid, just a single 500 GB > drive. I plan to buy more HDDs to have a bigger space for my > backups but since I cannot afford all HDDs at once I face a problem > of "expanding" an array. I'm able to add one 500 GB drive every few > months until I have all 4 drives. > > But I cannot make a backup of a backup... so reformatting/copying all > data each time when I add new disc to the array is not possible for me. > > Is it possible anyhow to create a "very degraded" raid array - a one > that consists of 4 drives, but has only TWO ? > > This would involve some very tricky *hole* management on the block > device... A one that places holes in stripes on the block device, > until more discs are added to fill the holes. When the holes are > filled, the block device grows bigger, and with lvm I just increase > the filesystem size. This is perhaps coupled with some "unstripping" > that moves/reorganizes blocks around to fill/defragment the holes. It's definitely not possible with raid5. Only option is to create a raid5 array consisting of less drives than it should contain at the end, and reshape it when you get more drives, as others noted in this thread. But do note the following points: o degraded raid5 isn't really Raid - i.e, it's not any better than a raid0 array, that is, any disk fails => the whole array fails. So instead of creating a degraded raid5 array initially, create smaller one instead, but not degraded, and reshape it when necessary. o reshaping takes time, and for this volume, reshape will take many hours, maybe days, to complete. o During this reshape time, errors may be fatal to the whole array - while mdadm do have a sense of "critical section", but the whole procedure isn't as much tested as the rest of raid code, I for one will not rely on it, at least for now. For example, a power failure at an "unexpected" moment, or some plain-stupid error in reshape code so that the whole array goes "boom" etc... o A filesystem on the array has to be resized separately after re{siz,shap}ing the array. And filesystems are different at this point, too - there are various limitations. For example, it's problematic to grow ext[23]fs by large amounts, because when it gets initially created, mke2fs calculates sizes of certain internal data structures based on the device size, and those structures can't be grown significantly, only recreating the filesystem will do the trick. > is it just a pipe dream? I'd say it is... ;) /mjt - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Janek Kozicki said: (by the date of Tue, 9 Oct 2007 00:25:50 +0200) > Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35 +1300) > > > No, but you can make a degraded 3 drive array, containing 2 drives and > > then add the next drive to complete it. > > > > The array can then be grown (man mdadm, GROW section), to add the fourth. > > Oh, good. Thanks, I must've been blind that I missed this. > This completely solves my problem. Uh, actually not :) My 1st 500 GB drive is full now. When I buy a 2nd one I want to create a 3-disc degraded array using just 2 discs, one of which contains unbackupable data. steps: 1. create degraded two-disc RAID5 on 1 new disc 2. copy data from old disc to new one 3. rebuild the array with old and new discs (now I have 500 GB on 2 discs) 4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2 discs) ... 5. when I buy 3rd drive I either grow the array, or just rebuild and wait with growing until I buy a 4th drive. Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible? PS: the fact, that degraded array will be unsafe for the data is an intented motivating factor for buying next drive ;) -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35 +1300) > No, but you can make a degraded 3 drive array, containing 2 drives and > then add the next drive to complete it. > > The array can then be grown (man mdadm, GROW section), to add the fourth. Oh, good. Thanks, I must've been blind that I missed this. This completely solves my problem. -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: very degraded RAID5, or increasing capacity by adding discs
} -Original Message- } From: [EMAIL PROTECTED] [mailto:linux-raid- } [EMAIL PROTECTED] On Behalf Of Richard Scobie } Sent: Monday, October 08, 2007 3:27 PM } To: linux-raid@vger.kernel.org } Subject: Re: very degraded RAID5, or increasing capacity by adding discs } } Janek Kozicki wrote: } } > Is it possible anyhow to create a "very degraded" raid array - a one } > that consists of 4 drives, but has only TWO ? } } No, but you can make a degraded 3 drive array, containing 2 drives and } then add the next drive to complete it. } } The array can then be grown (man mdadm, GROW section), to add the fourth. } } Regards, } } Richard I think someone once said you could create a 2 disk degraded RAID5 array with just 1 disk. Then add one later. Then expand as needed. Someone should test this. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
Janek Kozicki wrote: Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ? No, but you can make a degraded 3 drive array, containing 2 drives and then add the next drive to complete it. The array can then be grown (man mdadm, GROW section), to add the fourth. Regards, Richard - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: very degraded RAID5, or increasing capacity by adding discs
On Mon, 8 Oct 2007, Janek Kozicki wrote: Hello, Recently I started to use mdadm and I'm very impressed by its capabilities. I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine. The backup machine currently doesn't have raid, just a single 500 GB drive. I plan to buy more HDDs to have a bigger space for my backups but since I cannot afford all HDDs at once I face a problem of "expanding" an array. I'm able to add one 500 GB drive every few months until I have all 4 drives. But I cannot make a backup of a backup... so reformatting/copying all data each time when I add new disc to the array is not possible for me. Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ? This would involve some very tricky *hole* management on the block device... A one that places holes in stripes on the block device, until more discs are added to fill the holes. When the holes are filled, the block device grows bigger, and with lvm I just increase the filesystem size. This is perhaps coupled with some "unstripping" that moves/reorganizes blocks around to fill/defragment the holes. is it just a pipe dream? best regards PS: yes it's simple to make a degraded array of 3 drives, but I cannot afford two discs at once... -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html With raid1 you can create a degraded array with 1 disk- I have done this, I have always wondered if mdadm will let you make a degraded raid 5 array with 2 disks (you'd specify 3 and only give 2) - you can always expand later. Justin. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
very degraded RAID5, or increasing capacity by adding discs
Hello, Recently I started to use mdadm and I'm very impressed by its capabilities. I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine. The backup machine currently doesn't have raid, just a single 500 GB drive. I plan to buy more HDDs to have a bigger space for my backups but since I cannot afford all HDDs at once I face a problem of "expanding" an array. I'm able to add one 500 GB drive every few months until I have all 4 drives. But I cannot make a backup of a backup... so reformatting/copying all data each time when I add new disc to the array is not possible for me. Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ? This would involve some very tricky *hole* management on the block device... A one that places holes in stripes on the block device, until more discs are added to fill the holes. When the holes are filled, the block device grows bigger, and with lvm I just increase the filesystem size. This is perhaps coupled with some "unstripping" that moves/reorganizes blocks around to fill/defragment the holes. is it just a pipe dream? best regards PS: yes it's simple to make a degraded array of 3 drives, but I cannot afford two discs at once... -- Janek Kozicki | - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html