[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
You should always use 'nobootwait' on remote mounts so that the server can boot even when there is some problem with the remote mount. Drop in to #gluster on Freenode IRC and ping me, semiosis, for help troubleshooting this. Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
[Expired for mountall (Ubuntu) because there has been no activity for 60 days.] ** Changed in: mountall (Ubuntu) Status: Incomplete = Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
I create a bug (#1205075) about mountall that doesn't generate good mount commande line for glusterfs. My Bug was marked duplicate of this one. The final problem was the same, but the clues I found are pretty different. More, this bug has importance=low but for me it would be CRITICAL 'cause my server doesn't start anymore -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Hi guys, i have the exact same problem but with ocfs2. mountall tries to
mount my remote volume before o2cb is running, so it fails. There's a
really nasty workaround adding /etc/init.d/o2cb start in the mountall-
net.conf file, like this:
# mountall-net - Mount network filesystems
#
# Send mountall the USR1 signal to inform it to try network filesystems
# again.
description Mount network filesystems
start on net-device-up
task
script
/etc/init.d/o2cb start
PID=$(status mountall 2/dev/null | sed -e
'/start\/running,/{s/.*,[^0-9]*//;q};d')
[ -n $PID ] kill -USR1 $PID || true
end script
It works, but i'd like to avoid doing these things. I think that this
bug, is really close to this one
https://bugs.launchpad.net/ubuntu/+source/ocfs2-tools/+bug/474215?comments=all
, eventhough gluster an ocfs2 are different things.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1103047
Title:
mountall causes automatic mounting of gluster shares to fail
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
I'm getting this same behavior on Ubuntu 12.04 LTS with an ext4 filesystem (iSCSI volume). I am repeatedly seeing events like this sprinkled throughout my boot.log file. mount: special device /dev/mapper/mpath2-part1 does not exist mount: special device /dev/mapper/mpath3-part1 does not exist mount: special device /dev/mapper/mpath5-part1 does not exist mountall: mount /export/users [1133] terminated with status 32 mountall: mount /export/snapshots/users [1135] terminated with status 32 mountall: mount /export/faculty [1138] terminated with status 32 mount: special device /dev/mapper/mpath1-part1 does not exist mount: special device /dev/mapper/mpath4-part1 does not exist mountall: mount /export/snapshots/faculty [1140] terminated with status 32 mountall: mount /export/projects [1142] terminated with status 32 mount: special device /dev/mapper/mpath2-part1 does not exist mount: special device /dev/mapper/mpath3-part1 does not exist mount: special device /dev/mapper/mpath5-part1 does not exist mountall: mount /export/users [1175] terminated with status 32 mountall: mount /export/snapshots/users [1179] terminated with status 32 mountall: mount /export/faculty [1184] terminated with status 32 mount: special device /dev/mapper/mpath1-part1 does not exist mount: special device /dev/mapper/mpath4-part1 does not exist mount: special device /dev/mapper/mpath2-part1 does not exist mountall: mount /export/snapshots/faculty [1185] terminated with status 32 mountall: mount /export/projects [1192] terminated with status 32 mountall: mount /export/users [1201] terminated with status 32 mount: special device /dev/mapper/mpath3-part1 does not exist mountall: mount /export/snapshots/users [1208] terminated with status 32 mount: special device /dev/mapper/mpath5-part1 does not exist mountall: mount /export/faculty [1216] terminated with status 32 mount: special device /dev/mapper/mpath2-part1 does not exist mount: special device /dev/mapper/mpath5-part1 does not exist mount: special device /dev/mapper/mpath3-part1 does not exist mount: special device /dev/mapper/mpath1-part1 does not existmountall: mount /export/users [1265] terminated with status 32 mountall: mount /export/snapshots/users [1270] terminated with status 32 mountall: mount /export/faculty [1271] terminated with status 32 The offending lines in my /etc/fstab look like this: /dev/mapper/mpath2-part1 /export/users ext4 defaults,user_xattr,acl,barrier=1,errors=remount-ro,_netdev 0 0 /dev/mapper/mpath3-part1 /export/snapshots/usersext4 defaults,user_xattr,acl,barrier=1,errors=remount-ro,_netdev 0 0 /dev/mapper/mpath5-part1 /export/facultyext4 defaults,quota,user_xattr,acl,barrier=1,errors=remount-ro,_netdev 0 0 /dev/mapper/mpath1-part1 /export/snapshots/faculty ext4 defaults,user_xattr,acl,barrier=1,errors=remount-ro,_netdev 0 0 /dev/mapper/mpath4-part1 /export/projects ext4 defaults,quota,user_xattr,acl,barrier=1,errors=remount-ro,_netdev 0 0 As you can see, _netdev is specified and is being ignored. The filesystems do mount after the network is up, but not without lots of nasty errors. Here's the boot.log file with mountall set to verbose: Connected to Plymouth / is local /proc is virtual /sys is virtual /sys/fs/fuse/connections is virtual /sys/kernel/debug is virtual /sys/kernel/security is virtual /dev is virtual /dev/pts is virtual /tmp is local /run is virtual /run/lock is virtual /run/shm is virtual /boot is local /dev/mapper/quartz-swap_1 is swap /export/users is nowait /export/snapshots/users is nowait /export/faculty is nowait /export/snapshots/faculty is nowait /export/projects is nowait /home/projects is nowait mounting event sent for /sys/fs/fuse/connections mounting event sent for /sys/kernel/debug mounting event sent for /sys/kernel/security mounting event sent for /run/lock mounting event sent for /run/shm checking /boot fsck from util-linux 2.20.1 checking / fsck from util-linux 2.20.1 mounting event sent for swap /dev/mapper/quartz-swap_1 mounted event handled for /sys local 0/3 remote 0/0 virtual 1/10 swap 0/1 mounted event handled for /dev/pts local 0/3 remote 0/0 virtual 2/10 swap 0/1 mounting event handled for /sys/fs/fuse/connections mounting /sys/fs/fuse/connections mount /sys/fs/fuse/connections [352] exited normally mounting event handled for /sys/kernel/debug mounting /sys/kernel/debug mount /sys/kernel/debug [356] exited normally mounting event handled for /sys/kernel/security mounting /sys/kernel/security mount /sys/kernel/security [359] exited normally mounting event handled for /run/lock mounting /run/lock mount /run/lock [362] exited normally mounting event handled for /run/shm mounting /run/shm mount /run/shm [366] exited normally mounted event handled for /proc local 0/3 remote 0/0 virtual 3/10 swap 0/1 mounted event handled for /dev local 0/3 remote 0/0 virtual 4/10 swap 0/1 mounting event handled for swap /dev/mapper/quartz-swap_1 activating
Re: [Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
On Mon, Apr 15, 2013 at 11:07:10PM -, Zach Bethel wrote: As you can see, _netdev is specified and is being ignored. Incorrect. _netdev is being *respected*: _netdev does not tell the system anything about *which* network device a filesystem mount depends on, so mountall retries the mount after each network interface is brought up. The filesystems do mount after the network is up, but not without lots of nasty errors. That is, therefore, unrelated to this bug report. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Ok so the situation was as following. In boot.log we have: mounted event handled for /var/www local 2/3 remote 1/1 virtual 9/10 swap 0/0 Mount failed. Please check the log file for more details. mountall: mount /var/www [337] terminated with status 1 Filesystem could not be mounted: /var/www The thing is the log file mentionned, which I suppose to be the gluster log file, does not exist. Multiple attempts (and this one I do not explain) permitted me to have one time in boot.log a message stating that /va/log/glusterfs/var-www.log did not exist. Actually the folder exist, and permissions are fine as well. The only thing is that actually my /var/log is a dedicated partition like often on servers. I commented this out in fstab for a try, and managed to have the /va/log/glusterfs/var-www.log writtend at boot time. [2013-03-21 21:55:45.968441] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1 [2013-03-21 21:55:45.970384] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2013-03-21 21:55:45.970471] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host storage.domain.com [2013-03-21 21:55:45.970503] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success [2013-03-21 21:55:45.970523] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: 2 connect attempts left The fstab line is: storage.domain.com:web-g-www /var/wwwglusterfs defaults,_netdev,backupvolfile-server=172.18.2.130,fetch-attempts=1 0 0 I'm not sure where the 2 connect attempts left is coming from as I indicated fetch-attempts=1 is the fstab. Still we can see that DNS resolution is failing, may be indicating that the network is not really up when mountall is trying to mount glusterfs filesytems. Setting fetch-attempts=3 in the fstab managed to have all filesystem properly mounted, including glusterfs. So this tell us a few things: - network does not seem to be up when mountall try to mount glustersfs filesystems (at least DNS is failing) - there may be a missing dependency on having local fs up before trying to mount remote ones, because obviously the log file that could help in debugging this issue is not available for write when mountall (thus glusterfs) tries to write to it. (this is fixed by not separating /var/log but on a server may be dangerous then) Hope this helps -- Bruno -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Bruno, could you please attach the boot.log in question? I need to see the entirety of what mountall is doing to understand what's really going wrong. It's normal that mountall will attempt to mount all filesystems ASAP after boot, which means sometimes it tries to mount them before the network is up; then it will retry the mount for each network interface that's brought up. So seeing the full log will show which part isn't working correctly. One possibility is that glusterfs is taking so long to fail to mount, that between the time the mount fails and it actually notifies mountall that it /has/ failed, all the network interface events have already gone past. But The only thing is that actually my /var/log is a dedicated partition like often on servers. I don't think this is a very common configuration at all. I don't see any reason to have /var/log separate from the rest of /var. On the other hand, I'm not aware of any services that would currently have problems with this except for glusterfs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Hello, here is a log: http://pastebin.com/tBJLDVwj I did the mountall myself fro mthe command line (i.e not log from fstab). Might indeed be that glusterfs is passed the _netdev option and does not support it. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
From fstab we have this kind of log: http://pastebin.com/DBFPmRTQ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Bruno, your first pastebin shows: local 3/3 remote 0/1 virtual 11/11 swap 0/0 mounting event handled for /var/www mounting event sent for /var/www mounting event handled for /var/www unknown option _netdev (ignored) mounted event handled for /var/www remote finished This tells us several things: - using _netdev causes mountall to correctly identify /var/www as a remote mount - the _netdev option is passed to the mount command (which is probably a bug), and is not understood by the glusterfs mount helper, but is ignored and does *not* cause the mount to fail - when run after boot, mountall has no difficulty mounting glusterfs. But I really need to see a log of mountall from *boot* time, showing the failure to mount when using _netdev. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Just had the same thing happen on a number of our 12.04.2 KVM guests. The _netdev parameter is being ignored. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
I had to comment out the gluster fstab entry to get these 12.04.2 machines to finish their boot. Mountall kept throwing an mountall event failed before the system console was up. Adding and removing _netdev from the /etc/fstab had no affect. My workaround involved booting the KVMs on server ISO using recovery mode. Edited the fstab and added gluster mount to the rc.local file. sleep 3 mount.glusterfs [hostname]:/vol1 /mnt/glusterfs/vol1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Clayton, can you provide the output of 'mountall --verbose' requested in comment #5, when using the _netdev option? To get this output, add --verbose to the commandline in /etc/init/mountall.conf (as in the original bug report). BTW, glusterfs doesn't require any client daemons to be running for mounts to work, does it (like NFS does)? From the Ubuntu packages, it doesn't look like this is the case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
I am also running in this problem, and _netdev didn't fix it, and as the other guy said, right after booting, doing a `mount -a` mounts the /home just fine. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
I am also running in this problem, and _netdev didn't fix it Did you also get the unknown option _netdev (ignored) message? If so, when and where does it show up? _netdev is a standard option which should do exactly what's expected here. So if it's not working, I would need to see some logs from mountall --verbose to understand why. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Mounting of gluster volumes with or without the _netdev option in the fstab entry gives a: unknown option _netdev (ignored) It's my understanding that _netdev only works for nfs volumes, no? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
On Thu, Jan 24, 2013 at 10:19:03AM -, Bram De Wilde wrote: Mounting of gluster volumes with or without the _netdev option in the fstab entry gives a: unknown option _netdev (ignored) At what point do you get this message? It's my understanding that _netdev only works for nfs volumes, no? No, _netdev is a standard fstab option as described in the mount(8) manpage for declaring that the filesystem requires network access when this is not evident from the filesystem type. It's possible that mountall passes this through to the mount command and the glusterfs mount helper doesn't understand it, causing this warning message. But by itself this message shouldn't prevent the filesystem from mounting. Did the filesystems mount ok on boot after this change? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
mountall doesn't know that glusterfs is a network filesystem type. This means that at boot time, there's a race between bringing the network up and trying to mount /home, and when mountall wins the race, the mount fails and is never re-tried. The newer version of mountall is faster by mounting in parallel, which increases the chance of hitting this race and exposing this problem. To work around this bug, you should be able to add the '_netdev' option to the fstab entries, signal to mountall that the mount should be retried when network interfaces are brought up. Can you please test that this fixes the issue for you? ** Changed in: mountall (Ubuntu) Importance: Undecided = Low ** Changed in: mountall (Ubuntu) Status: New = Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1103047 Title: mountall causes automatic mounting of gluster shares to fail To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
