Re: [CentOS] raid1 custom initrd and yum
On Fri, 2008-04-04 at 00:54 -0500, Les Mikesell wrote: Sam Beam wrote: On Wednesday 02 April 2008 01:07, Les Mikesell wrote: First cut - in your recovery shell, comment out /home from /etc/fstab and see if you can come up without it (log in as root, of course). That will at least give you a fairly normal environment to try to figure out why the md1 device is getting assembled but the /dev/md1 node isn't created for it. Thanks Les, that was very helpful (and I should have thought of it...). But then it gets weird again: I commented out the /dev/md1 line and the system came all the way up to the login prompt. Great! I thought. Enter the root password and... kingkong login: root Password: xx Last login: Tue Dec 3 13:58:11 2002 /bin/bash: Permission denied doh! well of course I have done nothing special to the permissions there or anywhere else. I can see all the console boot messages and they all look normal. Booted into single user mode, and that works. /bin/bash has normal perms and all seems well. What's more, I was able to mount /dev/md1 on /home and it didn't complain. Then I un-commented the line in fstab, rebooted and it worked all the way up to the login prompt, it now uses all 3 md devices happily. But then, Permission denied is all I get. Nice system but it makes it hard to maintain when even root can't log in. Could there be some disk error? I have never had so much bizarre behavior from one system. Ready to chuck it out the window. Is there anything else I can try to see what's up? Is SELinux enabled? There's some black magic command to make it rebuild its labels when it is not happy. That would be touch ./autolable. -- ~/john OpenPGP Sig:BA91F079 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
Sam Beam wrote: On Wednesday 02 April 2008 01:07, Les Mikesell wrote: First cut - in your recovery shell, comment out /home from /etc/fstab and see if you can come up without it (log in as root, of course). That will at least give you a fairly normal environment to try to figure out why the md1 device is getting assembled but the /dev/md1 node isn't created for it. Thanks Les, that was very helpful (and I should have thought of it...). But then it gets weird again: I commented out the /dev/md1 line and the system came all the way up to the login prompt. Great! I thought. Enter the root password and... kingkong login: root Password: xx Last login: Tue Dec 3 13:58:11 2002 /bin/bash: Permission denied doh! well of course I have done nothing special to the permissions there or anywhere else. I can see all the console boot messages and they all look normal. Booted into single user mode, and that works. /bin/bash has normal perms and all seems well. What's more, I was able to mount /dev/md1 on /home and it didn't complain. Then I un-commented the line in fstab, rebooted and it worked all the way up to the login prompt, it now uses all 3 md devices happily. But then, Permission denied is all I get. Nice system but it makes it hard to maintain when even root can't log in. Could there be some disk error? I have never had so much bizarre behavior from one system. Ready to chuck it out the window. Is there anything else I can try to see what's up? Is SELinux enabled? There's some black magic command to make it rebuild its labels when it is not happy. -- Les Mikesell [EMAIL PROTECTED] ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
On Wednesday 02 April 2008 01:07, Les Mikesell wrote: First cut - in your recovery shell, comment out /home from /etc/fstab and see if you can come up without it (log in as root, of course). That will at least give you a fairly normal environment to try to figure out why the md1 device is getting assembled but the /dev/md1 node isn't created for it. Thanks Les, that was very helpful (and I should have thought of it...). But then it gets weird again: I commented out the /dev/md1 line and the system came all the way up to the login prompt. Great! I thought. Enter the root password and... kingkong login: root Password: xx Last login: Tue Dec 3 13:58:11 2002 /bin/bash: Permission denied doh! well of course I have done nothing special to the permissions there or anywhere else. I can see all the console boot messages and they all look normal. Booted into single user mode, and that works. /bin/bash has normal perms and all seems well. What's more, I was able to mount /dev/md1 on /home and it didn't complain. Then I un-commented the line in fstab, rebooted and it worked all the way up to the login prompt, it now uses all 3 md devices happily. But then, Permission denied is all I get. Nice system but it makes it hard to maintain when even root can't log in. Could there be some disk error? I have never had so much bizarre behavior from one system. Ready to chuck it out the window. Is there anything else I can try to see what's up? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
Sam Beam wrote on Tue, 1 Apr 2008 01:45:28 -0400: OK sorry to hammer the list but one more question - having almost got the drives mirrored and happy - since I have created a custom initrd that has the raid1 drivers in it, do I now have to tell yum to ignore kernel updates? Will the stock kernel render me unbootable? Is this about software RAID? You don't need to change the initrd in any way for this. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
On Tuesday 01 April 2008 05:31, Kai Schaetzl wrote: Sam Beam wrote on Tue, 1 Apr 2008 01:45:28 -0400: OK sorry to hammer the list but one more question - having almost got the drives mirrored and happy - since I have created a custom initrd that has the raid1 drivers in it, do I now have to tell yum to ignore kernel updates? Will the stock kernel render me unbootable? Is this about software RAID? You don't need to change the initrd in any way for this. ok (deep breathing relaxation exercise in order after staying up till 4AM trying different permutations)... All the howtos/posts I found on it seemed to this running mkinitrd to add raid1 was needed. So you don't run it at all after creating the md arrays? maybe that is why the system won't boot anymore after I synced the root partition? ;) I hope... ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
Sam Beam wrote: On Tuesday 01 April 2008 05:31, Kai Schaetzl wrote: Sam Beam wrote on Tue, 1 Apr 2008 01:45:28 -0400: OK sorry to hammer the list but one more question - having almost got the drives mirrored and happy - since I have created a custom initrd that has the raid1 drivers in it, do I now have to tell yum to ignore kernel updates? Will the stock kernel render me unbootable? Is this about software RAID? You don't need to change the initrd in any way for this. ok (deep breathing relaxation exercise in order after staying up till 4AM trying different permutations)... All the howtos/posts I found on it seemed to this running mkinitrd to add raid1 was needed. So you don't run it at all after creating the md arrays? If your system was installed without raid and you manually moved the / partition to an md device you would have to add the modules to the initrd yourself. However, subsequent yum updates that install a new kernel should include what you need - and they should not remove your running kernel or the supporting initrd, so you would have a fall-back from the grub boot menu anyway where you could fix it. maybe that is why the system won't boot anymore after I synced the root partition? ;) I hope... If you are booting a kernel that can't find your root partition, the initrd might be the problem. There are several other things that also have to be right. You should be able to boot your install cd/dvd with linux rescue at the boot prompt to fix any of them, so don't panic yet. -- Les Mikesell [EMAIL PROTECTED] ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
Sam Beam wrote on Tue, 1 Apr 2008 14:43:51 -0400: So you don't run it at all after creating the md arrays? I don't do any changes to initrd. I create the mirrored raid and I install grub on the second disk, so that it works for booting on both disks. The /boot partition needs to be on its own if you use LVM and not be LVM. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
On Tuesday 01 April 2008 15:03, Les Mikesell wrote: maybe that is why the system won't boot anymore after I synced the root partition? ;) I hope... If you are booting a kernel that can't find your root partition, the initrd might be the problem. There are several other things that also have to be right. You should be able to boot your install cd/dvd with linux rescue at the boot prompt to fix any of them, so don't panic yet. OK but can I panic if this system has a max of two IDE devices, no floppy, one PCI slot, and I don't have a PCI CD, and it won't boot from USB even? http://www.tyan.com/archive/products/html/gs10b2094_spec.html It has a hardware Promise RAID controller but I have it on good authority that I don't want to mess with that, software RAID is better, etc. I installed CentOS onto disk1 using the old 10-year old PC from the basement. Then I mirrored in the second disk. All went pretty well until the root partition was mirrored, fstab and /proc/mdstat and grub all agreed, and I rebooted. At this point Grub seems to work OK and the 3 RAID partitions (/boot, /home, /) assemble correctly and show UU. It gets all the way into INIT - set hostname, check for LVM, and then Checking filesystems and then tells me that /dev/md1 (should be /home) has a bad superblock: The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 device Then I am dropped to a recovery shell. but $ mdadm -E /dev/hd[ad]2 both show nice superblocks, that look OK to me. in dmesg I see raidautorun output and then device-mapper starting up as the last two entires But, /dev/md1 doesn't exist. /dev/md0 and /dev/md2 are there and seem normal. /proc/mdstat contains this: md1: active raid1 hdd2[1] hda2[0] 106012864 blocks [2/2] [UU] so imagine my surprise when I tried this: # mdadm -Q /dev/md1 mdadm: cannot open /dev/md1: No such file or directory # mdadm -Q /dev/hdd2 /dev/hdd2: is not an md array /dev/hdd2: device 1 in 2 device active raid1 /dev/.tmp.md1. Use mdadm --examine for more detail. But /dev/.tmp.md1 does not exist either, and therefore I can not stop this mystery array or fsck /dev/hd[ad]2 because they are busy being part of this non-existent device. Arrgh. I be stumped. Any help would be a big help ;) Sam ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] raid1 custom initrd and yum
Sam Beam wrote: I hope... If you are booting a kernel that can't find your root partition, the initrd might be the problem. There are several other things that also have to be right. You should be able to boot your install cd/dvd with linux rescue at the boot prompt to fix any of them, so don't panic yet. OK but can I panic if this system has a max of two IDE devices, no floppy, one PCI slot, and I don't have a PCI CD, and it won't boot from USB even? Not yet. http://www.tyan.com/archive/products/html/gs10b2094_spec.html It has a hardware Promise RAID controller but I have it on good authority that I don't want to mess with that, software RAID is better, etc. Promise makes a bunch of different stuff. I'd expect that you can run the drives separately - probably even hang a CD on it. I installed CentOS onto disk1 using the old 10-year old PC from the basement. Then I mirrored in the second disk. All went pretty well until the root partition was mirrored, fstab and /proc/mdstat and grub all agreed, and I rebooted. At this point Grub seems to work OK and the 3 RAID partitions (/boot, /home, /) assemble correctly and show UU. It gets all the way into INIT - set hostname, check for LVM, and then Checking filesystems and then tells me that /dev/md1 (should be /home) has a bad superblock: The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 device Then I am dropped to a recovery shell. but $ mdadm -E /dev/hd[ad]2 both show nice superblocks, that look OK to me. in dmesg I see raidautorun output and then device-mapper starting up as the last two entires But, /dev/md1 doesn't exist. /dev/md0 and /dev/md2 are there and seem normal. /proc/mdstat contains this: md1: active raid1 hdd2[1] hda2[0] 106012864 blocks [2/2] [UU] so imagine my surprise when I tried this: # mdadm -Q /dev/md1 mdadm: cannot open /dev/md1: No such file or directory If the device node doesn't exist, that makes sense. # mdadm -Q /dev/hdd2 /dev/hdd2: is not an md array /dev/hdd2: device 1 in 2 device active raid1 /dev/.tmp.md1. Use mdadm --examine for more detail. But /dev/.tmp.md1 does not exist either, and therefore I can not stop this mystery array or fsck /dev/hd[ad]2 because they are busy being part of this non-existent device. Arrgh. I be stumped. Any help would be a big help ;) First cut - in your recovery shell, comment out /home from /etc/fstab and see if you can come up without it (log in as root, of course). That will at least give you a fairly normal environment to try to figure out why the md1 device is getting assembled but the /dev/md1 node isn't created for it. I'm not sure what would happen if you create the node with the obvious major/minor numbers yourself. I'd try it, but don't blame me if it explodes. If you can't get /dev/md1 to appear, fdisk the partitions to type 83 so they won't even try to assemble. Then you can at least mount one of the underlying partitions to get to the data. Since you are this far along you probably don't need to pursue an alternate boot, but if you did, you could pull one of the drives and hang a cd or dvd on the ide interface. Raid1 will work just fine with one drive missing and you could sync the other one back later. -- Les Mikesell [EMAIL PROTECTED] ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] raid1 custom initrd and yum
OK sorry to hammer the list but one more question - having almost got the drives mirrored and happy - since I have created a custom initrd that has the raid1 drivers in it, do I now have to tell yum to ignore kernel updates? Will the stock kernel render me unbootable? Actually I am not 100% sure the initrd I created # mkinitrd -f --preload=raid1 /mnt/tmp/initrd-`uname -r`.img `uname -r` is even being used, because lsmod still lists a raid1 module. Wouldn't it be embedded in the running kernel and thus not in the module list? either way I am a little confused and this is important ;) best regards and thanks to all for the help, Sam ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
