Re: RAID array is gone, please help
First I'm getting the data off the RAID... then I'm going to delete the whole thing again... create a new RAID using partitions... follow every step carefully... then once the new RAID array is there, I'll throw a bit of data on it, and then reboot and see if it's still there... if so, I'll repopulate all the data. Thanks. On 3/23/2017 22:09, Lennart Sorensen wrote: On Thu, Mar 23, 2017 at 10:02:23PM +0100, Stephen Mueller wrote: Looks like it worked! Thanks! Well at least you could backup the data, just in case. I used: sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10 --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf I wonder about the assume-clean, but I would think any data you wrote will have been duplicated at the time you wrote it, and only unused space might not have been synced yet. Maybe running a forced resync would be worthwhile. And I got my instructions for creating the array here, and they also don't use partitions... https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04 Well it works, as long as you don't leave anything around to confuse it, in this case a GPT partition table that somehow came back to bite you. I wonder if mdadm could perhaps warn about the existing partition table when being asked to create a new device. Does it survive reboots now?
Re: RAID array is gone, please help
On Thu, Mar 23, 2017 at 10:02:23PM +0100, Stephen Mueller wrote: > Looks like it worked! Thanks! Well at least you could backup the data, just in case. > I used: > > sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10 > --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf I wonder about the assume-clean, but I would think any data you wrote will have been duplicated at the time you wrote it, and only unused space might not have been synced yet. Maybe running a forced resync would be worthwhile. > And I got my instructions for creating the array here, and they > also don't use partitions... > > https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04 Well it works, as long as you don't leave anything around to confuse it, in this case a GPT partition table that somehow came back to bite you. I wonder if mdadm could perhaps warn about the existing partition table when being asked to create a new device. Does it survive reboots now? -- Len Sorensen
Re: RAID array is gone, please help
On Thu, Mar 23, 2017 at 09:09:39PM +0100, Stephen Mueller wrote: > OK, I used gdisk to remove the GPT and MBR from each disk. > mdadm --assemble still doesn't work... says it can't find the > superblock. The mdadm --examine commands also say that no > superblock is detected. Yes the GPT overwrite the superblock. The GPT uses the first 17KB of the disk, and the superblock was at 4KB from the start. > I guess I'll go ahead with --create... At least given the data should start at 1MB from the start, it should not have been overwritten. I do always like disk images before I try anything like that. I am never quite sure what create will do although it will likely do the right thing at least if you get the order right. -- Len Sorensen
Re: RAID array is gone, please help
Looks like it worked! Thanks! I used: sudo mdadm --create /dev/md0 --assume-clean --verbose --level-10 --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf And I got my instructions for creating the array here, and they also don't use partitions... https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04 On 3/23/2017 21:09, Stephen Mueller wrote: OK, I used gdisk to remove the GPT and MBR from each disk. mdadm --assemble still doesn't work... says it can't find the superblock. The mdadm --examine commands also say that no superblock is detected. I guess I'll go ahead with --create... On 3/23/2017 20:59, Lennart Sorensen wrote: On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote: Apologies, I should have started this on linux-raid... stephen@fred> sudo gdisk -l /dev/sdc GPT fdisk (gdisk) version 1.0.1 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/sdc: 7814037168 sectors, 3.6 TiB Logical sector size: 512 bytes Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB Partition table holds up to 128 entries First usable sector is 34, last usable sector is 7814037134 Partitions will be aligned on 2048-sector boundaries Total free space is 7814037101 sectors (3.6 TiB) Number Start (sector)End (sector) Size Code Name stephen@fred> The other disks are all similar. How do I remove the MBR/GPT tables? So you would do that, and then the mdadm --create? Well at least that seems to confirm that something restored the GPT from the second copy at the end of the disk, wiping out the md superblock at 4k. If you run gdisk /dev/sdc, it has a 'zap' option using the z key to delete all traces of GPT. That ought to do what you want.
Re: RAID array is gone, please help
OK, I used gdisk to remove the GPT and MBR from each disk. mdadm --assemble still doesn't work... says it can't find the superblock. The mdadm --examine commands also say that no superblock is detected. I guess I'll go ahead with --create... On 3/23/2017 20:59, Lennart Sorensen wrote: On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote: Apologies, I should have started this on linux-raid... stephen@fred> sudo gdisk -l /dev/sdc GPT fdisk (gdisk) version 1.0.1 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/sdc: 7814037168 sectors, 3.6 TiB Logical sector size: 512 bytes Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB Partition table holds up to 128 entries First usable sector is 34, last usable sector is 7814037134 Partitions will be aligned on 2048-sector boundaries Total free space is 7814037101 sectors (3.6 TiB) Number Start (sector)End (sector) Size Code Name stephen@fred> The other disks are all similar. How do I remove the MBR/GPT tables? So you would do that, and then the mdadm --create? Well at least that seems to confirm that something restored the GPT from the second copy at the end of the disk, wiping out the md superblock at 4k. If you run gdisk /dev/sdc, it has a 'zap' option using the z key to delete all traces of GPT. That ought to do what you want.
Re: RAID array is gone, please help
On Thu, Mar 23, 2017 at 08:38:08PM +0100, Stephen Mueller wrote: > Apologies, I should have started this on linux-raid... > > > stephen@fred> sudo gdisk -l /dev/sdc > GPT fdisk (gdisk) version 1.0.1 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /dev/sdc: 7814037168 sectors, 3.6 TiB > Logical sector size: 512 bytes > Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB > Partition table holds up to 128 entries > First usable sector is 34, last usable sector is 7814037134 > Partitions will be aligned on 2048-sector boundaries > Total free space is 7814037101 sectors (3.6 TiB) > > Number Start (sector)End (sector) Size Code Name > stephen@fred> > > The other disks are all similar. > > How do I remove the MBR/GPT tables? So you would do that, and then the > mdadm --create? Well at least that seems to confirm that something restored the GPT from the second copy at the end of the disk, wiping out the md superblock at 4k. If you run gdisk /dev/sdc, it has a 'zap' option using the z key to delete all traces of GPT. That ought to do what you want. -- Len Sorensen
Re: RAID array is gone, please help
Apologies, I should have started this on linux-raid... stephen@fred> sudo gdisk -l /dev/sdc GPT fdisk (gdisk) version 1.0.1 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /dev/sdc: 7814037168 sectors, 3.6 TiB Logical sector size: 512 bytes Disk identifier (GUID): 60F1D54C-9D17-4688-BD6E-447F5E7408EB Partition table holds up to 128 entries First usable sector is 34, last usable sector is 7814037134 Partitions will be aligned on 2048-sector boundaries Total free space is 7814037101 sectors (3.6 TiB) Number Start (sector)End (sector) Size Code Name stephen@fred> The other disks are all similar. How do I remove the MBR/GPT tables? So you would do that, and then the mdadm --create? Thank you. On 3/23/2017 20:27, Lennart Sorensen wrote: On Thu, Mar 23, 2017 at 07:09:41PM +0100, r...@mueller.org wrote: Thank you very much or your reply. I naively thought that starting without partitions would be the best starting point, given 3 of the disks had been in a RAID5 array previously (possibly with partitions, not sure), but that looks like a bad choice, based on some other things I've googled. Lesson learned. I have an mdadm.conf file, but it could be a remnant of my previous array. I've already edited it trying to get things to work, so I'm not sure if it was updated when I created the new array or not. I see various people online have had success in my situation using madadm --create /dev/md0 --assume-clean --verbose --level=10 \ --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf Some people used --assume-clean, and some didn't. Given my array wasn't done with its resync, maybe I should leave that out. If that would work, I guess then I need to get the data off the array, delete it, and recreate it with disk partitions, or risk this happening again at the next reboot, for whatever reason. Anyone think it's a bad idea to try mdadm --create at this point? Sorry, I'm not sure how to write 0's to sector 0... Well the other possibility is that by having a MBR saying this has GPT and probably having mdadm corrupt the GPT by writing to 4k in, but nothing overwriting the backup GPT at the end of the device, something may have restored the GPT to the original location, even if it was an empty GPT, in which case your superblock is likely overwritten. Does 'gdisk -l /dev/sdc' claim that it is a valid GPT table or corrupt? So if that is what happened (not sure what would have done it, although it seems badly defined who is responsible for fixing the GPT if it is detected that the backup is valid but the primary is corrupt), then you will want to make sure the MBR and GPT tables are removed, before you do anything else. Likely the data is still there, since I think the first data block is offset a bit into the device, and the GPT is only the first 32KB or so of the device. If that is the case, doing the create would probably work. Of course the safest thing to do would be to clone your disks before trying to reassemble them (which means you need another four 4TB drives, but if the data is important, that's probably worth it). Of course you would also have to be sure the mdadm command you use to create it again is exactly the same as before, and that the device names are exactly the same as before.
Re: RAID array is gone, please help
On Thu, Mar 23, 2017 at 07:09:41PM +0100, r...@mueller.org wrote: > Thank you very much or your reply. > > I naively thought that starting without partitions would be the best > starting point, given 3 of the disks had been in a RAID5 array > previously (possibly with partitions, not sure), but that looks like > a bad choice, based on some other things I've googled. Lesson learned. > > I have an mdadm.conf file, but it could be a remnant of my previous array. > I've already edited it trying to get things to work, so I'm > not sure if it was updated when I created the new array or not. > > I see various people online have had success in my situation using > madadm --create /dev/md0 --assume-clean --verbose --level=10 \ > --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf > > Some people used --assume-clean, and some didn't. Given my array wasn't done > with its resync, maybe I should leave that out. > > If that would work, I guess then I need to get the data off the array, > delete it, and recreate it with disk partitions, or risk this happening > again at the next reboot, for whatever reason. > > Anyone think it's a bad idea to try mdadm --create at this point? > > Sorry, I'm not sure how to write 0's to sector 0... Well the other possibility is that by having a MBR saying this has GPT and probably having mdadm corrupt the GPT by writing to 4k in, but nothing overwriting the backup GPT at the end of the device, something may have restored the GPT to the original location, even if it was an empty GPT, in which case your superblock is likely overwritten. Does 'gdisk -l /dev/sdc' claim that it is a valid GPT table or corrupt? So if that is what happened (not sure what would have done it, although it seems badly defined who is responsible for fixing the GPT if it is detected that the backup is valid but the primary is corrupt), then you will want to make sure the MBR and GPT tables are removed, before you do anything else. Likely the data is still there, since I think the first data block is offset a bit into the device, and the GPT is only the first 32KB or so of the device. If that is the case, doing the create would probably work. Of course the safest thing to do would be to clone your disks before trying to reassemble them (which means you need another four 4TB drives, but if the data is important, that's probably worth it). Of course you would also have to be sure the mdadm command you use to create it again is exactly the same as before, and that the device names are exactly the same as before. -- Len Sorensen
Re: RAID array is gone, please help
Thank you very much or your reply. I naively thought that starting without partitions would be the best starting point, given 3 of the disks had been in a RAID5 array previously (possibly with partitions, not sure), but that looks like a bad choice, based on some other things I've googled. Lesson learned. I have an mdadm.conf file, but it could be a remnant of my previous array. I've already edited it trying to get things to work, so I'm not sure if it was updated when I created the new array or not. I see various people online have had success in my situation using madadm --create /dev/md0 --assume-clean --verbose --level=10 \ --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf Some people used --assume-clean, and some didn't. Given my array wasn't done with its resync, maybe I should leave that out. If that would work, I guess then I need to get the data off the array, delete it, and recreate it with disk partitions, or risk this happening again at the next reboot, for whatever reason. Anyone think it's a bad idea to try mdadm --create at this point? Sorry, I'm not sure how to write 0's to sector 0... Thank you. On 3/23/2017 18:37, Lennart Sorensen wrote: On Thu, Mar 23, 2017 at 05:49:05PM +0100, r...@mueller.org wrote: I am hoping someone here will help me. Was reading this site... https://raid.wiki.kernel.org/index.php/Linux_Raid and it said to email this list if you've tried everything other than mdadm --create. I am running Ubuntu 16.04. Machine name is fred. I used webmin to create a 4 disk RAID10 array yesterday. I moved all my data onto the array. Today, I had to reboot my PC. The resync was still not done, but I read online that it's OK to boot during resync. After boot, my array was gone. I checked syslog, and it just has this line: DeviceDisappeared event detected on md device /dev/md0 I did not partition my disks before building the array. So I believe the array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf. Here's some info... stephen@fred> lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 117.4G 0 disk ├─sda1 8:10 109.7G 0 part / ├─sda2 8:20 1K 0 part └─sda5 8:50 7.7G 0 part [SWAP] sdb 8:16 0 465.8G 0 disk └─sdb1 8:17 0 465.8G 0 part sdc 8:32 0 3.7T 0 disk sdd 8:48 0 3.7T 0 disk sde 8:64 0 3.7T 0 disk sdf 8:80 0 3.7T 0 disk stephen@fred> sudo mdadm --examine /dev/sdc [sudo] password for stephen: /dev/sdc: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdc1 mdadm: cannot open /dev/sdc1: No such file or directory stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdd /dev/sdd: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sde /dev/sde: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdf /dev/sdf: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd /dev/sde /dev/sdf mdadm: Cannot assemble mbr metadata on /dev/sdc mdadm: /dev/sdc has no superblock - assembly aborted Thank you for any help you can provide. Did your disks have partitions previously? That output looks a lot like the protective MBR partition table for a disk with GPT partitions. Could that still existing in sector 0 be confusing mdadm? I have never personally done any md raid without partitions. To me they just make more sense. One way to test could be to save a copy of sector 0, then overwrite sector 0 with zeros and then run mdadm --examine again to see if that makes a difference. You can always put back the saved copy of sector 0 that way. My understanding is that the default is to put the raid superblock at offset 4k, so it would not overwrite an existing MBR partition table. If it also happens due to rounding that the end of the disk isn't overwritten (or even just because that part of the filesystem wasn't written to yet), then the backup GPT from before would still be intact, and could perhaps cause even more confussion later if gdisk or similar is pointed at the disk. Really want to be sure there is no trace left of the partition table before using it raw for md raid. Any chance the system saved an mdadm.conf file of your setup?
Re: RAID array is gone, please help
On Thu, Mar 23, 2017 at 05:49:05PM +0100, r...@mueller.org wrote: > I am hoping someone here will help me. Was reading this site... > > https://raid.wiki.kernel.org/index.php/Linux_Raid > > and it said to email this list if you've tried everything other than mdadm > --create. > > > I am running Ubuntu 16.04. Machine name is fred. I used webmin to create a 4 > disk RAID10 array yesterday. I moved all my data onto the array. > > Today, I had to reboot my PC. The resync was still not done, but I read > online that it's OK to boot during resync. After boot, my array was gone. I > checked syslog, and it just has this line: > > DeviceDisappeared event detected on md device /dev/md0 > > I did not partition my disks before building the array. So I believe the > array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf. > > Here's some info... > > stephen@fred> lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:00 117.4G 0 disk > ├─sda1 8:10 109.7G 0 part / > ├─sda2 8:20 1K 0 part > └─sda5 8:50 7.7G 0 part [SWAP] > sdb 8:16 0 465.8G 0 disk > └─sdb1 8:17 0 465.8G 0 part > sdc 8:32 0 3.7T 0 disk > sdd 8:48 0 3.7T 0 disk > sde 8:64 0 3.7T 0 disk > sdf 8:80 0 3.7T 0 disk > > stephen@fred> sudo mdadm --examine /dev/sdc > [sudo] password for stephen: > /dev/sdc: >MBR Magic : aa55 > Partition[0] : 4294967295 sectors at1 (type ee) > stephen@fred> > stephen@fred> sudo mdadm --examine /dev/sdc1 > mdadm: cannot open /dev/sdc1: No such file or directory > stephen@fred> > stephen@fred> sudo mdadm --examine /dev/sdd > /dev/sdd: >MBR Magic : aa55 > Partition[0] : 4294967295 sectors at1 (type ee) > stephen@fred> > stephen@fred> sudo mdadm --examine /dev/sde > /dev/sde: >MBR Magic : aa55 > Partition[0] : 4294967295 sectors at1 (type ee) > stephen@fred> > stephen@fred> sudo mdadm --examine /dev/sdf > /dev/sdf: >MBR Magic : aa55 > Partition[0] : 4294967295 sectors at1 (type ee) > > stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd > /dev/sde /dev/sdf > mdadm: Cannot assemble mbr metadata on /dev/sdc > mdadm: /dev/sdc has no superblock - assembly aborted > > Thank you for any help you can provide. Did your disks have partitions previously? That output looks a lot like the protective MBR partition table for a disk with GPT partitions. Could that still existing in sector 0 be confusing mdadm? I have never personally done any md raid without partitions. To me they just make more sense. One way to test could be to save a copy of sector 0, then overwrite sector 0 with zeros and then run mdadm --examine again to see if that makes a difference. You can always put back the saved copy of sector 0 that way. My understanding is that the default is to put the raid superblock at offset 4k, so it would not overwrite an existing MBR partition table. If it also happens due to rounding that the end of the disk isn't overwritten (or even just because that part of the filesystem wasn't written to yet), then the backup GPT from before would still be intact, and could perhaps cause even more confussion later if gdisk or similar is pointed at the disk. Really want to be sure there is no trace left of the partition table before using it raw for md raid. Any chance the system saved an mdadm.conf file of your setup? -- Len Sorensen
RAID array is gone, please help
I am hoping someone here will help me. Was reading this site... https://raid.wiki.kernel.org/index.php/Linux_Raid and it said to email this list if you've tried everything other than mdadm --create. I am running Ubuntu 16.04. Machine name is fred. I used webmin to create a 4 disk RAID10 array yesterday. I moved all my data onto the array. Today, I had to reboot my PC. The resync was still not done, but I read online that it's OK to boot during resync. After boot, my array was gone. I checked syslog, and it just has this line: DeviceDisappeared event detected on md device /dev/md0 I did not partition my disks before building the array. So I believe the array consisted of /dev/sdc, /dev/sdd, /dev/sde, and /dev/sdf. Here's some info... stephen@fred> lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 117.4G 0 disk ├─sda1 8:10 109.7G 0 part / ├─sda2 8:20 1K 0 part └─sda5 8:50 7.7G 0 part [SWAP] sdb 8:16 0 465.8G 0 disk └─sdb1 8:17 0 465.8G 0 part sdc 8:32 0 3.7T 0 disk sdd 8:48 0 3.7T 0 disk sde 8:64 0 3.7T 0 disk sdf 8:80 0 3.7T 0 disk stephen@fred> sudo mdadm --examine /dev/sdc [sudo] password for stephen: /dev/sdc: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdc1 mdadm: cannot open /dev/sdc1: No such file or directory stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdd /dev/sdd: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sde /dev/sde: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> stephen@fred> sudo mdadm --examine /dev/sdf /dev/sdf: MBR Magic : aa55 Partition[0] : 4294967295 sectors at1 (type ee) stephen@fred> sudo mdadm --assemble --force /dev/md0 /dev/sdc /dev/sdd /dev/sde /dev/sdf mdadm: Cannot assemble mbr metadata on /dev/sdc mdadm: /dev/sdc has no superblock - assembly aborted Thank you for any help you can provide.