Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
> -Original Message- > From: centos-boun...@centos.org > [mailto:centos-boun...@centos.org] On Behalf Of Chan Chung > Hang Christopher > Sent: Saturday, August 15, 2009 10:20 > To: CentOS mailing list > Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery > > Robert Nichols wrote: > > Ross Walker wrote: > > > >> Since you don't know if LVM has a recovery path how can > you imply it > >> doesn't? > >> > > > > I've seen plenty of evidence that tools for LVM recovery > are lacking. > > I see postings from people asking about recovery of damaged LVM > > volumes and not getting any reasonable answers about how to repair. > > > This might help in future. > http://anthonydawson.thelasis.com/LVRecovery/index.htm > > > Or, complexity of dealing with things like duplicate volume group > > names when physically moving disk drives. Conventional > partitioning > > works just fine for me, and I'm confident that I can handle > just about > > any situation that crops up. I don't need an additional layer of > > complexity. > > > > > Which is why I do not use default names when creating > volgroups and logvols. I always found it handy to use ASSET#_VG#_MMDDHHMM as a "default" pattern here. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100- - +1 (443) 269-1555 x333Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Robert Nichols wrote: > Ross Walker wrote: > >> Since you don't know if LVM has a recovery path how can you imply it >> doesn't? >> > > I've seen plenty of evidence that tools for LVM recovery are lacking. > I see postings from people asking about recovery of damaged LVM > volumes and not getting any reasonable answers about how to repair. > This might help in future. http://anthonydawson.thelasis.com/LVRecovery/index.htm > Or, complexity of dealing with things like duplicate volume group names > when physically moving disk drives. Conventional partitioning works > just fine for me, and I'm confident that I can handle just about any > situation that crops up. I don't need an additional layer of > complexity. > > Which is why I do not use default names when creating volgroups and logvols. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Lanny Marcus wrote: > On Fri, Aug 14, 2009 at 8:03 AM, Kristopher > Kane wrote: > I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D >> Maybe you could point him to this list for lunch time lesson reading, >> however, you won't be able to talk about him behind his back anymore. >> > > If your former employer is going to continue to employ him in that > position, possibly he should be using webmin to do sys admin? That > might prevent some disaster(s). Or, at least, he might get a warning > from webmin, before he destroys something. > Not possible...he has to access a postgresql server via console to run sql commands or fix up some other access. Likewise with vpopmail admin. He has to add new disks and (sigh) move the vpopmail stuff at the very least over to the filesystems. You guys can help him if he gets himself on the list. Doing final data checking on the disks. I got the partition layout wrong in the end but I have that sorted now. Got both md devices back online and pvscan checked out during the rebuild of the md array that is the basis for the pv so I guess things will be okay later too. Just got a final fsck to run on the logvols and I am tossing this back in his hands. (Bored yet?) Things have not turned out as bad as I thought they might have. Ah well, at least I know there is way to check lvm metadata now. Link here just in case someone can point out any errors as I have not quite have to opportunity to put the information to the test. http://anthonydawson.thelasis.com/LVRecovery/index.htm ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Kristopher Kane wrote: >>> I get to learn something new at his expense, (which is now just a scare) >>> nice successor eh? :-D >>> > > Maybe you could point him to this list for lunch time lesson reading, > however, you won't be able to talk about him behind his back anymore. > Haha, I am not worried about that at all. I will point him over here. He should have clear up the piled up work (Windows related, gah) by now. Not naming him but please be good to him...he may not be up on mailing list etiquette and other stuff like RTM. No, I ain't teaching him all that. I have babysat him for long enough. > :-/ 8-| ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Ross Walker wrote: > > Since you don't know if LVM has a recovery path how can you imply it > doesn't? I've seen plenty of evidence that tools for LVM recovery are lacking. I see postings from people asking about recovery of damaged LVM volumes and not getting any reasonable answers about how to repair. Or, complexity of dealing with things like duplicate volume group names when physically moving disk drives. Conventional partitioning works just fine for me, and I'm confident that I can handle just about any situation that crops up. I don't need an additional layer of complexity. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
On Aug 14, 2009, at 9:22 PM, Robert Nichols wrote: > Ross Walker wrote: >> On Aug 14, 2009, at 12:48 PM, Robert Nichols >> wrote: >> >>> Ross Walker wrote: On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopher wrote: Question now is, was the first sector of partition 1 damaged (was it 63 or 64 sectors dd'd)? If so it will require a more tricky procedure to fix. >>> No, the ext2 file system does not use the first 1K block of the >>> partition. >>> That space is left free for a boot loader. The first super block >>> of the >>> file system is the 2nd 1K block of the partition, and even if that >>> gets >>> overwritten it is easily recovered from one of the backup super >>> blocks. >> >> I did not know that, good information. >> >> So stage2 is written in first 1k of the partition? > > No, nowhere near enough room. Just stage 1 goes into the first > sector. > The second sector is unused. Stage 1 then loads the fs-specific > stage 1.5 > from absolute sector numbers on the disk. Stage 1.5 then loads > stage 2 > from the file system. (Yes, if the *_stage1_5 files get moved on the > disk, you have to re-run the GRUB installer.) > > The only time more than the 446-byte stage 1 gets installed is when > installing GRUB in the MBR or in the boot sector of a logical drive in > the extended partition. There, there can be a full track available, > which is enough room for the appropriate stage 1.5, but not for the > 100+ KB of stage 2. > >> Ah but I believe the OP has the partition setup as a LVM PV. Question >> is does LVM leave the first 1k free for a stage2 boot loader? >> >> And if the LVM metadata is corrupt is there a second copy at the end >> of the PV? > > No idea. I don't use LVM. Having a known recovery path if things get > munged is more important to me than the features of LVM. Since you don't know if LVM has a recovery path how can you imply it doesn't? -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
On Fri, Aug 14, 2009 at 8:03 AM, Kristopher Kane wrote: >>> I get to learn something new at his expense, (which is now just a scare) >>> nice successor eh? :-D > > Maybe you could point him to this list for lunch time lesson reading, > however, you won't be able to talk about him behind his back anymore. If your former employer is going to continue to employ him in that position, possibly he should be using webmin to do sys admin? That might prevent some disaster(s). Or, at least, he might get a warning from webmin, before he destroys something. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Ross Walker wrote: > On Aug 14, 2009, at 12:48 PM, Robert Nichols > wrote: > >> Ross Walker wrote: >>> On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang >>> Christopher wrote: >>> >>> Question now is, was the first sector of partition 1 damaged (was it >>> 63 or 64 sectors dd'd)? >>> >>> If so it will require a more tricky procedure to fix. >> No, the ext2 file system does not use the first 1K block of the partition. >> That space is left free for a boot loader. The first super block of the >> file system is the 2nd 1K block of the partition, and even if that gets >> overwritten it is easily recovered from one of the backup super blocks. > > I did not know that, good information. > > So stage2 is written in first 1k of the partition? No, nowhere near enough room. Just stage 1 goes into the first sector. The second sector is unused. Stage 1 then loads the fs-specific stage 1.5 from absolute sector numbers on the disk. Stage 1.5 then loads stage 2 from the file system. (Yes, if the *_stage1_5 files get moved on the disk, you have to re-run the GRUB installer.) The only time more than the 446-byte stage 1 gets installed is when installing GRUB in the MBR or in the boot sector of a logical drive in the extended partition. There, there can be a full track available, which is enough room for the appropriate stage 1.5, but not for the 100+ KB of stage 2. > Ah but I believe the OP has the partition setup as a LVM PV. Question > is does LVM leave the first 1k free for a stage2 boot loader? > > And if the LVM metadata is corrupt is there a second copy at the end > of the PV? No idea. I don't use LVM. Having a known recovery path if things get munged is more important to me than the features of LVM. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
On Aug 14, 2009, at 12:48 PM, Robert Nichols wrote: > Ross Walker wrote: >> On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang >> Christopher wrote: >> >> Question now is, was the first sector of partition 1 damaged (was it >> 63 or 64 sectors dd'd)? >> >> If so it will require a more tricky procedure to fix. > > No, the ext2 file system does not use the first 1K block of the > partition. > That space is left free for a boot loader. The first super block of > the > file system is the 2nd 1K block of the partition, and even if that > gets > overwritten it is easily recovered from one of the backup super > blocks. I did not know that, good information. So stage2 is written in first 1k of the partition? Ah but I believe the OP has the partition setup as a LVM PV. Question is does LVM leave the first 1k free for a stage2 boot loader? And if the LVM metadata is corrupt is there a second copy at the end of the PV? -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Ross Walker wrote: > On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang > Christopher wrote: > > Question now is, was the first sector of partition 1 damaged (was it > 63 or 64 sectors dd'd)? > > If so it will require a more tricky procedure to fix. No, the ext2 file system does not use the first 1K block of the partition. That space is left free for a boot loader. The first super block of the file system is the 2nd 1K block of the partition, and even if that gets overwritten it is easily recovered from one of the backup super blocks. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
> -Original Message- > From: centos-boun...@centos.org > [mailto:centos-boun...@centos.org] On Behalf Of Ross Walker > Sent: Friday, August 14, 2009 10:30 > To: CentOS mailing list > Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery > > On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang > Christopher wrote: > > > >> > >> First of all, I would dd a copy of the whole drive off to > another drive, so you can have a few goes at this. > >> > >> How do you know only those bits where lost? > >> > > > > The dd command zeros the first 64 sectors, that is, the mbr > and then > > the next 63 sectors which would the bootsector of the first > partition > > and the next 62 sectors following that. The first partition on both > > disks belong to the md device that is the basis for the physical > > volume for the system. And if it had not, it would have belonged to > > the md device for the /boot partition which is not a great loss. > > Default Redhat layout this. > > > > The box is still live and I am glad he was not clueless > enough to say > > yes to the mkefs2 command he was following from whatever > howto he had > > been looking at. It looks like the lvm survived having the first 62 > > sectors being zeroed. Apparently lvm uses the first 255 > sectors/blocks > > for lvm config data. No alarms/warnings in the logs. Certainly no > > panics otherwise I would not even be able to get in. > > > > I get to learn something new at his expense, (which is now just a > > scare) nice successor eh? :-D > > > > Absolutely zero data loss. What can I say? > > Well if he dd'd 64 sectors instead of 63 then the first > sector of the first partition is going to be zero'd too. > > Backup the data while the stale partition table is still in memory! > > A reboot will make it inaccessible. > > Try a: > > # sfdisk -d /dev/sdX >/root/sdX.save ROTFLMAO, you should make sure, before copying and pasting that /root/sdX.save does not exist. > > Look at it and see if it contains a valid partition table, if > it does then do a: > > # sfdisk --no-reread /dev/sdX > Question now is, was the first sector of partition 1 damaged (was it > 63 or 64 sectors dd'd)? > > If so it will require a more tricky procedure to fix. > > -Ross > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos > -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100- - +1 (443) 269-1555 x333Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
On Fri, Aug 14, 2009 at 5:00 AM, Chan Chung Hang Christopher wrote: > >> >> First of all, I would dd a copy of the whole drive off to another drive, so >> you can have a few goes at this. >> >> How do you know only those bits where lost? >> > > The dd command zeros the first 64 sectors, that is, the mbr and then the > next 63 sectors which would the bootsector of the first partition and > the next 62 sectors following that. The first partition on both disks > belong to the md device that is the basis for the physical volume for > the system. And if it had not, it would have belonged to the md device > for the /boot partition which is not a great loss. Default Redhat layout > this. > > The box is still live and I am glad he was not clueless enough to say > yes to the mkefs2 command he was following from whatever howto he had > been looking at. It looks like the lvm survived having the first 62 > sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks > for lvm config data. No alarms/warnings in the logs. Certainly no panics > otherwise I would not even be able to get in. > > I get to learn something new at his expense, (which is now just a scare) > nice successor eh? :-D > > Absolutely zero data loss. What can I say? Well if he dd'd 64 sectors instead of 63 then the first sector of the first partition is going to be zero'd too. Backup the data while the stale partition table is still in memory! A reboot will make it inaccessible. Try a: # sfdisk -d /dev/sdX >/root/sdX.save Look at it and see if it contains a valid partition table, if it does then do a: # sfdisk --no-reread /dev/sdX http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
>> I get to learn something new at his expense, (which is now just a scare) >> nice successor eh? :-D Maybe you could point him to this list for lunch time lesson reading, however, you won't be able to talk about him behind his back anymore. :-/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
- Original Message > From: Chan Chung Hang Christopher > To: CentOS mailing list > Sent: Friday, 14 August, 2009 10:00:41 > Subject: Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery > > > > > > First of all, I would dd a copy of the whole drive off to another drive, so > you can have a few goes at this. > > > > How do you know only those bits where lost? > > > > The dd command zeros the first 64 sectors, that is, the mbr and then the > next 63 sectors which would the bootsector of the first partition and > the next 62 sectors following that. The first partition on both disks > belong to the md device that is the basis for the physical volume for > the system. And if it had not, it would have belonged to the md device > for the /boot partition which is not a great loss. Default Redhat layout > this. > > The box is still live and I am glad he was not clueless enough to say > yes to the mkefs2 command he was following from whatever howto he had > been looking at. It looks like the lvm survived having the first 62 > sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks > for lvm config data. No alarms/warnings in the logs. Certainly no panics > otherwise I would not even be able to get in. > > I get to learn something new at his expense, (which is now just a scare) > nice successor eh? :-D > > Absolutely zero data loss. What can I say? > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos I think if you can access the data at the moment, I would back it up and set the LVM stuff up and mbr up again, unless you are really sure you can fix the zero'd info. Either way, if the data is accessible, back it up now! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
> > First of all, I would dd a copy of the whole drive off to another drive, so > you can have a few goes at this. > > How do you know only those bits where lost? > The dd command zeros the first 64 sectors, that is, the mbr and then the next 63 sectors which would the bootsector of the first partition and the next 62 sectors following that. The first partition on both disks belong to the md device that is the basis for the physical volume for the system. And if it had not, it would have belonged to the md device for the /boot partition which is not a great loss. Default Redhat layout this. The box is still live and I am glad he was not clueless enough to say yes to the mkefs2 command he was following from whatever howto he had been looking at. It looks like the lvm survived having the first 62 sectors being zeroed. Apparently lvm uses the first 255 sectors/blocks for lvm config data. No alarms/warnings in the logs. Certainly no panics otherwise I would not even be able to get in. I get to learn something new at his expense, (which is now just a scare) nice successor eh? :-D Absolutely zero data loss. What can I say? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
- Original Message > From: Chan Chung Hang Christopher > To: CentOS mailing list > Sent: Friday, 14 August, 2009 3:31:32 > Subject: [CentOS] OT: Fortunate clueless dd chum - lvm recovery > > Looks like the chum did not have to lose any data. > > Wiping out the MBR and the next 63 blocks apparently only wiped out grub > stage1, partition table, and part of the lvm config data. > > > I get to try to do a lvm 'recovery' at his expense now but this is my > first time...has anybody ever tried restoring lvm config data back to > the md/pv device? > > > Cheers, > > Christopher > ___ > CentOS mailing list > CentOS@centos.org > http://lists.centos.org/mailman/listinfo/centos First of all, I would dd a copy of the whole drive off to another drive, so you can have a few goes at this. How do you know only those bits where lost? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Johnny Hughes wrote: > Chan Chung Hang Christopher wrote: > >> Chan Chung Hang Christopher wrote: >> >>> Looks like the chum did not have to lose any data. >>> >>> >> I cannot believe he actually tried to create a new filesystem on sda >> according to the .bash_history file after the dd commands. I think I >> need a titanium clueby4. Anybody know where I can get one? >> > > Shouldn't be a problem, just recover from the backup right. Surely he > does backups :D > > Nope...he is stilling working on that one... XD The company fried the disks for the previous backup server solution shortly after I left. I kept telling them that they should first fix up the new 'server room' before anything else for the office renovation. No, the boss would not listen. Under powering the disks (external disk cage connected via eSata) due to too many devices on the same power socket did the rest. Good thing I left before that disaster. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Chan Chung Hang Christopher wrote: > Chan Chung Hang Christopher wrote: >> Looks like the chum did not have to lose any data. >> > I cannot believe he actually tried to create a new filesystem on sda > according to the .bash_history file after the dd commands. I think I > need a titanium clueby4. Anybody know where I can get one? Shouldn't be a problem, just recover from the backup right. Surely he does backups :D signature.asc Description: OpenPGP digital signature ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT: Fortunate clueless dd chum - lvm recovery
Chan Chung Hang Christopher wrote: > Looks like the chum did not have to lose any data. > I cannot believe he actually tried to create a new filesystem on sda according to the .bash_history file after the dd commands. I think I need a titanium clueby4. Anybody know where I can get one? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
