[CentOS] filesystem mounting fails at boot
Hi I have an IBM blade with internal harddisks, in hardware RAID1. I've installed a CentOS 6 64bit on it, everything works just fine. After the installation, I've presented a vdisk to the blade from an external SAN (an HP EVA4000), connected through FiberChannel. I've partitioned the disk, formatted it and mounted it under /store, then added it to fstab. Everything was fine, until I rebooted. At boot I'm getting the following error for /store: Mounting local filesystems: mount: special device UUID=2a587e95-4a6c-4336-bb8b-f0d066905bc5 does not exist It just goes on to boot without mounting this filesystem. After it boots, I can log in and give the command "mount -a", and it gets mounted without problems. As far as I can tell, the reason for this is that CentOS doesn't wait for the external disk to get initialized fully and it just doesn't find it at boot time. I have other CentOS blades, installed and booting from the same SAN and they work without problems, but I noticed that they wait a little bit longer at boot. Am I missing some stuff from initrd? What can I do to make it wait for the block device a bit longer before it tries mounting it ? Thanks! -- Imre Gergely http://havaz.net gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
Does it work usin netdev option? Eero 10.10.2015 4.17 ip. "Imre Gergely" kirjoitti: > > Hi > > I have an IBM blade with internal harddisks, in hardware RAID1. I've > installed a CentOS 6 64bit on it, everything works just fine. > > After the installation, I've presented a vdisk to the blade from an > external SAN (an HP EVA4000), connected through FiberChannel. I've > partitioned the disk, formatted it and mounted it under /store, then > added it to fstab. Everything was fine, until I rebooted. > > At boot I'm getting the following error for /store: > > Mounting local filesystems: mount: special device > UUID=2a587e95-4a6c-4336-bb8b-f0d066905bc5 does not exist > > It just goes on to boot without mounting this filesystem. After it > boots, I can log in and give the command "mount -a", and it gets mounted > without problems. > > As far as I can tell, the reason for this is that CentOS doesn't wait > for the external disk to get initialized fully and it just doesn't find > it at boot time. I have other CentOS blades, installed and booting from > the same SAN and they work without problems, but I noticed that they > wait a little bit longer at boot. > > Am I missing some stuff from initrd? What can I do to make it wait for > the block device a bit longer before it tries mounting it ? > > Thanks! > > -- > Imre Gergely > http://havaz.net > gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
_netdev The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system). This device is not a network device (this a SAN not a NAS). To the OS it looks like a normal SCSI attached device, it's /dev/sdb. In the blade there is a HBA (Qlogic) card, and it's connected through FiberChannel. If I understand these terms correctly, it has nothing to do with the network. On 10/10/2015 04:20 PM, Eero Volotinen wrote: > Does it work usin netdev option? > > Eero > 10.10.2015 4.17 ip. "Imre Gergely" kirjoitti: > >> Hi >> >> I have an IBM blade with internal harddisks, in hardware RAID1. I've >> installed a CentOS 6 64bit on it, everything works just fine. >> >> After the installation, I've presented a vdisk to the blade from an >> external SAN (an HP EVA4000), connected through FiberChannel. I've >> partitioned the disk, formatted it and mounted it under /store, then >> added it to fstab. Everything was fine, until I rebooted. >> >> At boot I'm getting the following error for /store: >> >> Mounting local filesystems: mount: special device >> UUID=2a587e95-4a6c-4336-bb8b-f0d066905bc5 does not exist >> >> It just goes on to boot without mounting this filesystem. After it >> boots, I can log in and give the command "mount -a", and it gets mounted >> without problems. >> >> As far as I can tell, the reason for this is that CentOS doesn't wait >> for the external disk to get initialized fully and it just doesn't find >> it at boot time. I have other CentOS blades, installed and booting from >> the same SAN and they work without problems, but I noticed that they >> wait a little bit longer at boot. >> >> Am I missing some stuff from initrd? What can I do to make it wait for >> the block device a bit longer before it tries mounting it ? >> >> Thanks! >> >> -- >> Imre Gergely >> http://havaz.net >> gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 >> >> ___ >> CentOS mailing list >> CentOS@centos.org >> https://lists.centos.org/mailman/listinfo/centos >> > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos > -- Imre Gergely http://havaz.net gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
On October 10, 2015 8:34:11 AM CDT, Imre Gergely wrote: > > _netdev > The filesystem resides on a device that requires network >access (used to prevent the system from attempting to mount these >filesystems until the network has been enabled on the system). > >This device is not a network device (this a SAN not a NAS). To the OS >it >looks like a normal SCSI attached device, it's /dev/sdb. In the blade >there is a HBA (Qlogic) card, and it's connected through FiberChannel. >If I understand these terms correctly, it has nothing to do with the >network. > >On 10/10/2015 04:20 PM, Eero Volotinen wrote: >> Does it work usin netdev option? >> >> Eero >> 10.10.2015 4.17 ip. "Imre Gergely" kirjoitti: >> >>> Hi >>> >>> I have an IBM blade with internal harddisks, in hardware RAID1. I've >>> installed a CentOS 6 64bit on it, everything works just fine. >>> >>> After the installation, I've presented a vdisk to the blade from an >>> external SAN (an HP EVA4000), connected through FiberChannel. I've >>> partitioned the disk, formatted it and mounted it under /store, then >>> added it to fstab. Everything was fine, until I rebooted. >>> >>> At boot I'm getting the following error for /store: >>> >>> Mounting local filesystems: mount: special device >>> UUID=2a587e95-4a6c-4336-bb8b-f0d066905bc5 does not exist >>> >>> It just goes on to boot without mounting this filesystem. After it >>> boots, I can log in and give the command "mount -a", and it gets >mounted >>> without problems. >>> >>> As far as I can tell, the reason for this is that CentOS doesn't >wait >>> for the external disk to get initialized fully and it just doesn't >find >>> it at boot time. I have other CentOS blades, installed and booting >from >>> the same SAN and they work without problems, but I noticed that they >>> wait a little bit longer at boot. >>> >>> Am I missing some stuff from initrd? What can I do to make it wait >for >>> the block device a bit longer before it tries mounting it ? >>> Does dmesg provide any useful information? If you remove (assuming they are there) rhgb and quiet and add debug to the kernel line at boot time does that give you any more info? How does the dmesg output compare to one that is able to mount the disk at boot without issue? Barry ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
On 10/10/2015 05:03 PM, Barry Brimer wrote: > On October 10, 2015 8:34:11 AM CDT, Imre Gergely wrote: >> _netdev >> The filesystem resides on a device that requires network >> access (used to prevent the system from attempting to mount these >> filesystems until the network has been enabled on the system). >> >> This device is not a network device (this a SAN not a NAS). To the OS >> it >> looks like a normal SCSI attached device, it's /dev/sdb. In the blade >> there is a HBA (Qlogic) card, and it's connected through FiberChannel. >> If I understand these terms correctly, it has nothing to do with the >> network. >> >> On 10/10/2015 04:20 PM, Eero Volotinen wrote: >>> Does it work usin netdev option? >>> >>> Eero >>> 10.10.2015 4.17 ip. "Imre Gergely" kirjoitti: >>> Hi I have an IBM blade with internal harddisks, in hardware RAID1. I've installed a CentOS 6 64bit on it, everything works just fine. After the installation, I've presented a vdisk to the blade from an external SAN (an HP EVA4000), connected through FiberChannel. I've partitioned the disk, formatted it and mounted it under /store, then added it to fstab. Everything was fine, until I rebooted. At boot I'm getting the following error for /store: Mounting local filesystems: mount: special device UUID=2a587e95-4a6c-4336-bb8b-f0d066905bc5 does not exist It just goes on to boot without mounting this filesystem. After it boots, I can log in and give the command "mount -a", and it gets >> mounted without problems. As far as I can tell, the reason for this is that CentOS doesn't >> wait for the external disk to get initialized fully and it just doesn't >> find it at boot time. I have other CentOS blades, installed and booting >> from the same SAN and they work without problems, but I noticed that they wait a little bit longer at boot. Am I missing some stuff from initrd? What can I do to make it wait >> for the block device a bit longer before it tries mounting it ? > Does dmesg provide any useful information? If you remove (assuming they are > there) rhgb and quiet and add debug to the kernel line at boot time does that > give you any more info? How does the dmesg output compare to one that is able > to mount the disk at boot without issue? > Yes, I tried that, and look at dmesg, but couldn't find anything interesting. The only useful info is that the drive is not ready when it tries to mount it, but it becomes ready a little bit later. The one that is mounting the disk without issue is a bit different because it also boots from the SAN so it really needs to have the external storage available from the beginning. This one boots from internal disk and only later wants to mount the external disk. -- Imre Gergely http://havaz.net gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
> On Oct 10, 2015, at 9:34 AM, Imre Gergely wrote: > _netdev > The filesystem resides on a device that requires network > access (used to prevent the system from attempting to mount these > filesystems until the network has been enabled on the system). _netdev in fstab was a workaround from oracle linux support for a FC issue very similar to the one you described(OS/driver may not plogi in to the storage quickly enough.). At least give it try before so quickly dismissing assistance. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
On 10/10/2015 10:06 PM, Steven Tardy wrote: >> On Oct 10, 2015, at 9:34 AM, Imre Gergely wrote: >> _netdev >> The filesystem resides on a device that requires network >> access (used to prevent the system from attempting to mount these >> filesystems until the network has been enabled on the system). > _netdev in fstab was a workaround from oracle linux support for a FC issue > very similar to the one you described(OS/driver may not plogi in to the > storage quickly enough.). At least give it try before so quickly dismissing > assistance. > ___ > You are right of course. I'm sorry. I will try it tonight and let you know. -- Imre Gergely http://havaz.net gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] filesystem mounting fails at boot
On 10/10/2015 11:41 AM, Imre Gergely wrote: The one that is mounting the disk without issue is a bit different because it also boots from the SAN ...which means that the HBA driver is included in the initrd, but not in the system where you're having trouble. Edit /etc/dracut.conf.d/hba.conf and add one line: add_drivers+="your_hba_driver" ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
[CentOS] Clustering and ha planning
Hello , Centos users: My name is Leandro, I have been using Centos for 4 years and this is the first post in this mail list. I would like to study and introduce myself in clustering and high availability for Centos, currently I have not experience at all about it. I would like to ask about the newest method to achieve high availability , load balancing on linux / Centos. So far I have seen the Clustering docs writen for Centos 5 and the HA documentation from www.linux-ha.org that have been published in 2010. So, I would like to ask to comunity, which are the new methods for clustering and get HA and where to get updated documentation. Regards, Leandro. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Clustering and ha planning
Hello Leandro, CentOS 5 is quite old and different from current CentOS 7, some things have changed, mostly improved and as usual your favourite search engine is your friend. e.g. http://clusterlabs.org/quickstart-redhat.html https://skcave.wordpress.com/2014/11/04/creating-high-availability-cluster-with-centos-7/ -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Leandro" > To: centos@centos.org > Sent: Saturday, 10 October, 2015 22:06:38 > Subject: [CentOS] Clustering and ha planning > Hello , Centos users: > My name is Leandro, I have been using Centos for 4 years and this is the > first post in this mail list. > I would like to study and introduce myself in clustering and high > availability for Centos, currently I have not experience at all about it. > I would like to ask about the newest method to achieve high availability > , load balancing on linux / Centos. > So far I have seen the Clustering docs writen for Centos 5 and the HA > documentation from www.linux-ha.org that have been published in 2010. > So, I would like to ask to comunity, which are the new methods for > clustering and get HA and where to get updated documentation. > > Regards, > Leandro. > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Clustering and ha planning
On 10/10/2015 2:06 PM, Leandro wrote: So, I would like to ask to comunity, which are the new methods for clustering and get HA and where to get updated documentation. I contend the appropriate approach to HA should be based on what services you need to keep available. an HA file server has quite different requirements and implementations than a HA relational database server or a HA DNS server. There's no magic one size fits all solutions. -- john r pierce, recycling bits in santa cruz ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Clustering and ha planning
Thanks for pointing that. I would like to learn about clustering and HA, so if I have to chose a service for my testing scenario It will be a radius or a mysql justo to keep it simple. Leandro. On 10/10/15 18:49, John R Pierce wrote: On 10/10/2015 2:06 PM, Leandro wrote: So, I would like to ask to comunity, which are the new methods for clustering and get HA and where to get updated documentation. I contend the appropriate approach to HA should be based on what services you need to keep available. an HA file server has quite different requirements and implementations than a HA relational database server or a HA DNS server. There's no magic one size fits all solutions. ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Clustering and ha planning
The main mailing list for HA clustering in "Clusterlabs Users": http://clusterlabs.org/mailman/listinfo/users. It's not strictly for any OS, but RHEL/CentOS and SUSE are probably the most common OSes. I might recommend starting with this: https://alteeve.ca/w/History_of_HA_Clustering The Linux-HA project (heartbeat) is long deprecated. The stack to learn is Corosync + Pacemaker. As Nux mentioned, CentOS 7 is the best platform. As you'll see in the History link above, there was a lot of changes that happened between 2008 ~ 2013 era. Learning on any older CentOS means you're learning an old stack, which probably doesn't make sense outside of a few cases. We also have an active IRC channel on #clusterlabs on freenode.net, too. If you stop by, be sure to idle. Folks are from all over so different people are around at different times. That said, people are good about replying to questions when they come around. Welcome to HA! digimer On 10/10/15 05:06 PM, Leandro wrote: > Hello , Centos users: > My name is Leandro, I have been using Centos for 4 years and this is the > first post in this mail list. > I would like to study and introduce myself in clustering and high > availability for Centos, currently I have not experience at all about it. > I would like to ask about the newest method to achieve high availability > , load balancing on linux / Centos. > So far I have seen the Clustering docs writen for Centos 5 and the HA > documentation from www.linux-ha.org that have been published in 2010. > So, I would like to ask to comunity, which are the new methods for > clustering and get HA and where to get updated documentation. > > Regards, > Leandro. > > ___ > CentOS mailing list > CentOS@centos.org > https://lists.centos.org/mailman/listinfo/centos -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] problem on exceptional quit
I am not sure if we can not send attachments to the mailing list. There were quite a lot replies before, but I got nothing back since attachements was added. I will remove the attachments and send it again. Please have a look at the email below. Thanks for your help. --- Dear All, Thanks for all your help. I will put all the comments together. Please have a look if there is any clue on such ghost problem. I have also attached the log files: dmesg, secure, messages. Please note that there is a message in secure when it exited just now. Oct 9 10:55:55 maya2012 su: pam_unix(su:session): session closed for user root > Can you trigger the error reliably by doing something network intensive, like > scp or rsync a large file? I've seen similar behaviour with a bad NIC that > was in the process of dying. Yes, I copied tens of Gb files using rsync. It worked well. > That's very often a result of IP conflict. I'm assuming that you're > connecting to an IPv4 address. If so, log in to your CentOS server and use > arping to look for conflicts: > > # arping -c 2 D -I em1 The IP is fixed to my server. The network administrator has checked the address, and only this computer uses it. When I run the above command line, the output is: [root@maya2012 hwang]# arping -c 2 -D -I em1 222.200.125.5 ARPING 222.200.125.5 from 0.0.0.0 em1 Sent 2 probes (2 broadcast(s)) Received 0 response(s) >> 1. Login via Mac, Windows, Linux systems from different computers. >> 2. Modify sshd_config on the server as suggested by many posts: >> TCPKeepAlive yes >> ClientAliveInterval 60 > > TCPKeepAlive is "yes" by default. ClientAliveInterval doesn't appear to be a > valid setting. Either TCPKeepAlive or ServerAliveInterval could be useful if > the problem were a stateful firewall which was dropping your connection from > its state table, and then resetting the connection in response to a later > packet from your client. > > Since those don't help, that tends to suggest that the problem isn't an > intermediate host, but the server itself. Possibly an IP conflict. Also, > check the output of "dmesg" to see if there are any problems recorded with > the NIC. Check the output of "ifconfig" to see if there are TX or RX errors > that increase when your connections are reset. [root@maya2012 hwang]# ifconfig em1: flags=4163 mtu 1500 inet 222.200.125.5 netmask 255.255.255.128 broadcast 222.200.125.127 inet6 fe80::d6ae:52ff:fe6a:405e prefixlen 64 scopeid 0x20 ether d4:ae:52:6a:40:5e txqueuelen 1000 (Ethernet) RX packets 2865 bytes 396191 (386.9 KiB) RX errors 0 dropped 180 overruns 0 frame 0 TX packets 510 bytes 55844 (54.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 em2: flags=4099 mtu 1500 ether d4:ae:52:6a:40:5f txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 0 (Local Loopback) RX packets 7 bytes 748 (748.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 7 bytes 748 (748.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@maya2012 hwang]# ip -s -d l l 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 RX: bytes packets errors dropped overrun mcast 74870 0 0 0 TX: bytes packets errors dropped carrier collsns 74870 0 0 0 2: em1: mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether d4:ae:52:6a:40:5e brd ff:ff:ff:ff:ff:ff promiscuity 0 RX: bytes packets errors dropped overrun mcast 312908 2272 0 138 0 1081 TX: bytes packets errors dropped carrier collsns 43946 403 0 0 0 0 3: em2: mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000 link/ether d4:ae:52:6a:40:5f brd ff:ff:ff:ff:ff:ff promiscuity 0 RX: bytes packets errors dropped overrun mcast 0 00 0 0 0 TX: bytes packets errors dropped carrier collsns 0 00 0 0 0 Thanks, Hua ___ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
